I think gear is almost all upside. I spent many years riding in no gear, or just a helmet. Wearing proper gear is so much more comfortable and makes riding better for me. I never knew how much jeans sucked to ride in until I bought some riding pants. The safety is just a bonus.
Downside? Getting all geared up takes time. I used to think it took a long time to get geared up for the summer. Now I dream of summer when I don't need glove liners, balaclava, thermal underwear, etc.
I get excited putting the gear on. It gives me time to go over everything mentally. Think about the upcoming ride, remember something I was about to forget, finish my coffee, whatever.
I have never, ever, ever had anyone make fun of me or felt odd because I was geared up. I hear this all the time: People saying they feel like a dork in their gear. Get over yourself. Nobody gives a shit about you or what you a wearing. People are way too self-involved.