Even though I haven't ridden one, I think you picked a great bike to start out on. It's not too light so it should be stable on-road, not too heavy so it'll handle the rough stuff and be easier to pick up, should be geared to cruise the road all day.
If you're buying handguards, I'd get these. They cost more but you want the ones that mount to the triple tree to save room on your handlebars. You'll run out of room later on.
Call rckymtn and have them send you their 2013 catalog. Its nice to refer to.
To make you feel better, the 1st time I took my KLR out it went about the same. It was a little windy in town, but when I got out in the wide open it was very windy. I was trying to stay in the main/smooth track but the wind kept blowing me onto the side in the loose stuff. The bike still had the stock/junk tires and I wasn't use to riding a buffalo, so it was more of a stressful ride instead of a fun, relaxing ride.
Now I can blast along on gravel doing 65mph while standing.
You'll get there!!