I also had no difficulty with road bikes on gravel roads. I picked up a cheap dual-sport and it really is nicer on gravel roads and I've ridden ATV trails here where my road bikes would never go.
You're not asking about the "right bike" but I'd now say that lighter is more playful. (Mine is heavier and more road-worthy.) I also vote for drop-ability. I generally like bikes that I don't have to worry about.
But my main vote is "try it, you'll like it".
"The belief that there is only one truth, and that oneself is in possession of it, is the root of all evil in the world."-- Max Born, Nobel Physicist