For anyone who's ever been to Glacier National Park in Montana, there is a Forest Service road called North Fork Road, which is where my dad and I happened to end up on two rather unsuitable bikes. Last summer, my dad and I visited Glacier, me on my ZRX1100 and him on his GL1500. We arrived at the park a little late and all of the front-end, easily accessible campsites were full. We pulled out the map of the park and located a campsite at Bowman Lake, and headed for it, not knowing what the roads there and back would be like.
Now, that road is probably not terribly challenging on a smaller DS bike, and probably not even that bad on a bigger ADV bike with some ground clearance, but it was quite an experience for us - me on a 120hp street bike with slick tires and next to no ground clearance, and him on his 900+lb behemoth. It was one of those experiences that was fun in retrospect but at the time we were both constantly worrying about being able to make it up a gravelly hill, or bottoming out and cracking an oil pan, leaving us stranded miles from help.
I would have much preferred to have been on a ~300lb DS bike and would have had a lot more fun, too. We did pass a group of riders (going the other way, we'd never have been able to overtake anyone) on KLRs and DR650s and such who were clearly having more fun than us.
The smaller bike is almost always going to be better suited to rougher conditions, and that's what I'd take in a heartbeat. The biggest bike I'd want to be riding on anything rougher than a graded gravel road is a 650.