I love VFFs and other minimalist shoes for hiking. When I first started wearing them my feet would get a little sore on rocky trails but now I can wear them on any terrain for any length hike.
Sorry, can't attest to climbing though. It probably depends on the durability of the upper material (a lot of them are very thin fabric that would snag and rip pretty easily, at least on real rock...not sure about the gym holds). There's a style called the Trek that is now now discontinued, but would be a good option if you can find them on sale or used somewhere. Also, since the toes are separate, if you're standing on a really small hold I imagine most or all of your weight would be on your big toe, rather than distributed across the entire foot. Since you've been running barefoot for a long time maybe your feet are strong enough that this isn't an issue, but it sounds uncomfortable to me!
But yes, minimalist shoes in general are awesome and you should get a pair!