I've searched, since I first saw this thread, for where my aversion to high viz comes from, and finally figured it out.
To me, while I wear a helmet everywhere, at minimum kevlar Rider's pants and an armored jacket, gloves and boots, there is a limit to how much safety I'm willing to do.
Let me explain. I grew up in the 60's and 70's, and when we were kids we climbed trees, made ramps for our Stingrays with no helmets, rode brakeless minibikes in jeans, tee shirts, and All Stars as fast as they would go, made go karts and hauled ass down hilly streets, rode motorcycles helmetless, bombed down hills on skateboards helmetless, etc. etc.
You don't see that any more. A kid making a ramp for his or her bike? Without a helmet? Their parents, although they did the same thing as kids, would take the bike away. You don't see kids in trees any more, and they ride skateboards only at parks with helmets, knee pads, elbow pads and parental supervision. Everything is about safety, to the point that it becomes rediculous and even detremental to human life. When I was a kid, no one had diabetes. Sure, there were a few fat kids, but for the most part they kept up with us playing football, riding bikes, etc.
This is where the high vis comes in for me. It is screaming that motorcycles are dangerous! To me, when I see someone in high vis gear on a motorcycle, I see someone who is so averse to risk that they probably are so worried about getting hit they aren't enjoying themselves. I also believe this sends the same message to the public, who already believe motorcycles are deadly, do deadly they would rather actually kill their kids than to see them on one.
It's just that one step towards the bull^%$t ovwer-safety conscious attitude I am unwilling to take.
Wear your favorite color jacket, pants if you got 'em, your favorite color helmet, and ride like they can't see you. And have fun. The end.