Try a Dr/xr/klr. They're all pretty good do all bikes. The klr is the heaviest with best seat,the Dr/xr are more dirt bike than the klr though. I gave the Dr a nod due to cheaper build up cost and more aftermarket parts. Husky makes a 610,BMW has a 650,and ktm has a 640/690. Those are all in about the same class. If you're just doing gravel then you can go bigger without issues,but I wouldn't recommend a bigger bike for off road or single track. Even at single track the 650's are a bit big. Just match your bike to your riding. Your smaller bikes will be less highway friendly if you commute on one any distance, but much nicer off road. A drz400 might do you well too. Id say it's between the current 250s and 650's in terms of road/street.
Your smaller bikes would be a wr250r,klx250,Honda's 230,and a host of others. The wr being heads above the others.
If you just do gravel don't worry so much about a dual sport. Most bikes can handle gravel,just mind the paint and chrome.
Most bigger "ADV" bikes are good for forest roads and the like. I wouldn't recommend a big 1200gs off road,but given patience and skill it's possible. Just remember,she'll fall hard..and she's heavy to pick up. The same for running a plated dirt bike on the highway. Possible,but you really wind one up on the slab. Look into a plate conversion if you do lots of off road and barely any highway, they're legal in most states.