For me the two downsides to ATGATT is
1) Less Riding: There have been many times when I wanted to take the bike out but didn't because I was too lazy to switch the clothes I was in and change to ATG. Also I don't want to be carrying all my gear when I arrive at my destination. Minor inconveniance but enough of one to make me take the car instead of the bike.
2) Sweat and heat: For the first couple of years at my job I used to commute everyday to work on my bike before I had a car, 10 miles of traffic lights and intersections. I would get to work with huge sweat spots and it was annoying to carry gear and change of clothes when the parking lot is three blocks away from your office. Now that I have a car I'll only take the bike to work when the temp is lower than 85 and dry or 75 and humid.
Even after reading the face plant posts and knowing about all the various gear that is available I am still not atgatt.
As for what I wear and when:
Everytime I ride: Full face helmet, gloves and motorcycle specific boots.
Highway : Same as above and motorcycle jacket with armor
Town: Same as above on V-Strom, 50/50 jacket/no jacket on Shadow cruiser.
Never ride with shorts, sneakers, or short sleeves.
No riding Pants as of yet - deciding now on what to get depending on conveniance of use and application. Did recently purchase a pair of strap on T-Pro knee pads. Figure at least my knees will be protected from impact and the strap on feature let me put on/off with whatever pants I'm wearing without having to take off my shoes.