You're right but it depends how quickly and comfortably you want to do it. I swapped from a Scrambler to my XC and it's chalk and cheese. The longer travel suspension of a proper "adventure" bike makes the dirt sections so much easier. If fact I also prefer the XC on the tarmac.
2002 black Tiger955i, 72000 miles and counting.
2012 black Tiger800XC, 32000 miles and counting.