Almost all the time. Even in winter I tend to kick the shoes off and drive in socks.
Don't find any problems at all with feet slipping, as I can wrap my toes around the pedals I'd say it's less likely for me. Better feel too.
Only car that i don't like driving barefoot much is my wifes and that because the pedals seem to be a long way from the floor. Have to have my foot at quite a sharp angle between foot and shin. It would probably work well if I started wearing heels.