You're right but it depends how quickly and comfortably you want to do it. I swapped from a Scrambler to my XC and it's chalk and cheese. The longer travel suspension of a proper "adventure" bike makes the dirt sections so much easier. If fact I also prefer the XC on the tarmac.
2002 black Tiger955i, 71000 miles.
2012 black Tiger800XC, 26000 miles.
Bike history [(N) = bought new] :- YAS1, RD350B(N), XS750(N), XT500, XJ650(N), XJ900S(N), CBX750, XL185S, XR250RF, XR600, KTM600LC4, TDR250, R100GS, TS200R, Tiger955i(N), Scrambler900efi, Tiger800XC(N)...