The road across from the glider port is Lost Valley Road and the campground is Indian Flats, (7 mi. in). Campground is closed at the end of Feb.for about 5 mos. for some kind of endangered frog!
Also, IIRC Blake mentions the road across from the Warner Springs glider port (Lost Creek?) which is a favorite of mine. The first few miles are paved jeep trail being very narrow & twisty... also little traffic and scenic.
I haven't yet taken that road over the ridge to the north.
There's a nice, unimproved campground right where the pavement ends as the road crosses the ridge (Lost Indian Campground?)
Palm Springs and the Cahuilla desert is just a little farther north (as is Joshua Tree) but I don't know of a short way down.