I feel boatload more peer pressure to wear gear than I do to go without. Maybe if I rode a cruiser or a Harley the pressure would go the other way? I think a ton of people wear gear to fit in, not necessarily because of safety.
And it doesn't bother me either way. I'll wear what I feel comfortable and safe in.