I also had no difficulty with road bikes on gravel roads. I picked up a cheap dual-sport and it really is nicer on gravel roads and I've ridden ATV trails here where my road bikes would never go.
You're not asking about the "right bike" but I'd now say that lighter is more playful. (Mine is heavier and more road-worthy.) I also vote for drop-ability. I generally like bikes that I don't have to worry about.
But my main vote is "try it, you'll like it".
Motorcycles are magical.