I agree with those that mention the heavy weight of the KLR. I think that's the main reason it is slow compared to other 650 singles mentioned. The power to weight ratio is poor.
I've owned all of the aforementioned bikes and honestly, the KLR does not feel down on power related to the other bikes. I even owned the XRL and the KLR at the same time. The XRL is not more powerful. It is more abrupt and has a cruder more dirt bike like engine character, whereas, the KLR is much smoother and thereby makes it feel slower, but it's not.
I have a 2000 model and have stripped it down to a reasonable level, but could go much further if I wanted. I estimate I've taken between 20-30 lbs off, but really it's still going to be much heavier than the stock XRL or DR.
Now having said that, I've been searching and researching performance mods for the motor that are reasonable and practical. (By my estimation) Things that can be done that fall in this category are as follows: (in no particular order, but done together) Porting, aftermarket cams, bigger valves with better springs, the 688cc (Wyman) kit, and lighten the flywheel, and switch out the carb. IIRC, the compression ratio on the stock motor is acceptable, somewhere around 11:1, so I wouldn't try decking the head or anything else to bump up the compression, although you could if you wanted to. Then you need to make some modifications to the cooling system to compensate.
Patman and KLRCary did great work, but that was all done 5 years ago and since they were pioneers, others have seemingly built upon their work, but the documentation is spotty.
Food for thought.