You nailed. That's the dilemma.
You didn't say what type of road riding you plan to do. If you are just doing short sections of highway/back roads to access trails or go to work, the XT would be great. It will be a lot more fun off road, and let you gain confidence quickly.
If you plan to spend 4 hours on the highway at 70 mph, the XT will be a humbling experience. The KLR can ride all day at those speeds and more, even with a passenger. Off road, however, light and medium trails will probably be the limit. Sure there are some daredevils that can really off road the KLR, but most started on smaller bikes.
If you are looking at new, have you considered the DR650? It is smaller physically, and easier to man handle. My off road bike is a 1983 XL600R that is roughly the same size as the DR. I am about your size, and find that size bike good for everything. I wouldn't get too extreme, but when I get in a jam, I can drag the bike off the hill if needed. It will also do 70+ all day. 2-up riding is a strain on it, but doable.