In general I would say no.
The main reason I say this is in first place it's not too hard for BRD to get close to fast cast cap using equipment alone.
Also, if you're not in the situation where you're pulling in a merit party, fast cast normally isn't -really- important..it's still nice to have but not worth sacrificing the extra support you get with /WHM (-na, Erase, Curaga, Divine Seal, and the extra MP which actually makes a big difference for a BRD).
I guess it depends what exactly you mean by "healing isn't a problem." If you literally are never required to lift a finger in the healing department then I suppose you may as well /RDM but I can't think of any situation where you consistently never need to heal at all.
BRD is all about supporting your party and /RDM just doesn't have much to offer here. Most of the time if a BRD isn't healing it's because they're lazy and not because they shouldn't be.
Edit: if you still need to know, Fast Cast I+II will get you 15% reduced casting time and 7.5% reduced recast.