I think the real potential downside is that the changes to your riding style are subtle and largely subconscious. If you don't believe me, ride around the block with no gear and see for yourself how cautiously you ride. Most people definitely ride differently with gear on, and I would venture to say that no one rides the same with gear as they would without. So the question isn't "does the gear cause you to ride in a riskier manner?", but rather "how much?".
Again, I'm not arguing against wearing gear, just trying to point out some of the perhaps not so obvious psychology underneath.