Many of the D-37 desert racers use the CRFX as a base. Having done it myself allow me to add a couple of things:
The R does not have the magic E button.
The R close ratio gear box is too close for desert usage (and assuming Rally Raid).
Many guys use R cams in the X head and there are different tuned cams that come on different years X & R.
Not uncommon for guys to swap the X head for the R head.
The R uses a 19" rear wheel, the X uses a 18".
Not sure about the newer bikes in the last couple of years, but the at one time the X had larger radiators.
I know that you can't directly fit an R pipe onto a X, how much of that is due to the different shrouds, tank size, or if the mounts on the frames are different or what.
If it was me I would start with the X, and I'm pretty sure that what JCR starts with for their Baja bikes.
"I couldn't wait for success, so I went ahead without it."