The xkcd comic Time Management gets at one aspect of this issue. The tolltip caption is the important part:
I never trust anyone who's more excited about success than about doing the thing they want to be successful at.
So true. It's not that success doesn't matter, but are you driven by success or by doing something and trying to be successful with it? If success is the driver, then I am somewhat leery. If you have a passion and want to be successful with it, then that is totally cool.