I nearly always wear a dark tint shield, but usually use the shield more as a sunshade than visor. My main rides have big fairings and windshields so normally I just wear prescription sunglasses on sunny days. . I carry a clear shield in my trunk and will switch out if it rains.
I have two main helmets, an HJC and Arai. Switching out a shield on the Arai is a nightmare--I don't know what they were thinking, but it sounds like you are breaking all kinds of stuff in there when you put a shield on or take one off.