I'd forget buying the "perfect" bike, and just get something you can afford and get out there. Buy used, learn from the experience, have some fun.
I was where you are a few years ago, though I had some some Jurassic-era dirt experience.
In the last 3 years, I have had a '99 DR350SE, '06 DR650SE, and now an '09 WR250R. Hasn't cost much to play, all sold for close to what I had in them. Anything with a plate that lets you get out there should do for a starting point. As someone noted above, in general lighter will make you smile more.
KLX250S, WR250R, DR-Z400, XR650L, DR650SE, whatever. KLR if you are gravel road biased, not so much if dirt means single track.
For a cheap starting point, with your inseam, I'd look for either a KLX or DR650. DR is better on the road, KLX on the trail, both do both.
They're cheap, plentiful, reliable, and no matter what you buy, it will be your first dual-sport but probably not your last.