I'll recommend incorporating the Palomar Divide Truck Trail (on the northeast side of Palomar off 79) as there's less traffic, more of the sort of dirt we're looking for and the route is interesting & scenic. Starting/ending in Temecula for this.
Also, IIRC Blake mentions the road across from the Warner Springs glider port (Lost Creek?) which is a favorite of mine. The first few miles are paved jeep trail being very narrow & twisty... also little traffic and scenic.
I haven't yet taken that road over the ridge to the north.
There's a nice, unimproved campground right where the pavement ends as the road crosses the ridge (Lost Indian Campground?)
Palm Springs and the Cahuilla desert is just a little farther north (as is Joshua Tree) but I don't know of a short way down.