You're right but it depends how quickly and comfortably you want to do it. I swapped from a Scrambler to my XC and it's chalk and cheese. The longer travel suspension of a proper "adventure" bike makes the dirt sections so much easier. If fact I also prefer the XC on the tarmac.
2002 black Tiger955i, problem free 72000 miles.
2012 black Tiger800XC, problem free 28000 miles.