Well, in my opinion it is not worth it unless you are really devoted to chess.. for instance consider the following:
- I am an e4-player which was a choice based on taste, though I sometimes feel uneasy with some openings e.g. the Caro-Kann.
- Learning the whole d4-complex would mean, say 100% of studying.
- (seriously) Learning the Caro-Kann Advance alone would mean only 5-10% of studying.
- Learning an anti-Caro-Kann line, e.g. KIA, would mean 2% of studying.
- Learning a completely off-beat line, e.g. Larsen (1.b3), Sokolsky (1.b4), or even 1.f3 (better choose something that does not transpose a lot

), would mean only 1% of studying.
Plus: - Your opponents will most probably have a pet line against both e4 & d4. E.g. say I play 1.d4 to avoid the Caro-Kann and my opponent replies with the Slav! (a bit familiar isn't it?

).
- If you participate in tournaments etc., you will need at least a year to get ready a full repertoire based on 1.d4. On the other hand losing every now and then a game to Caro-Kann looks manageable.
- A good balance of time spend on openings / tactics / strategy-middlegame / endings / practice / other (you choose the correct mix of course) is important to improve and also have fun.
To summarise:
- I don't think your opponents will be put off by you playing 1.d4 (in the example above), hence mostly a waste of time.
- For me not playing 1.d4 involves a dislike for Nimzos, QID and the rest. It is much easier to learn (and learn to love!) the Caro-Kann.
- It is much easier to use these off-beat and anti- lines especially if you choose the right opponent/timing (e.g. blitz games).
P.S.: The only really good reasons I see in playing both 1.e4 & 1.d4 is a) you may learn new types of positions & maybe change/enhance your style, b) if you are a really serious player and tend to know your opponents well, you may be able to prepare something nasty for them (e.g. Fischer-Spassky), or c) you are a chess coach.
regards, chk