You're right but it depends how quickly and comfortably you want to do it. I swapped from a Scrambler to my XC and it's chalk and cheese. The longer travel suspension of a proper "adventure" bike makes the dirt sections so much easier. If fact I also prefer the XC on the tarmac.
2002 black Tiger955i, 72000 miles and counting.
2012 black Tiger800XC, The other British Land Rover. 40000 miles and counting.
2013 black Tiger800XC, The other British Land Rover. 500 miles and counting.