You can't pick and choose when it is "safe" and when it is "not safe". People sometimes die in low-speed crashes (remember the motor cop that just died in a funeral procession?) and people sometimes walk away from high-speed crashes.
I used to be 90% ATGATT but since it's warming up I've started taking the 3-mile ride to work wearing just my work uniform (khakis and a polo), a helmet, gloves, and boots. Is it slightly more dangerous? Yes. Is it much more convenient, comfortable, and most of all liberating? YES.