I was riding down to the video rental place on my 10-speed last summer, and dumped the bike. I have one speed on a bicycle and that's flat out. I was wearing a tank top and jeans, I hit my head and roached my shoulder / forearm / hip / knee pretty badly, I was wearing a helmet (thank god) but my shoulder looked like bacon for a month. I estimate I was doing 20ish km/h.
The guy that I bought my VFR from crashed his 50cc Vino in a parking lot at less than 30km/h, he caught the front wheel on a tar strip and ended up breaking his shoulder blade (dunno what he was wearing). It healed well but afterwards he found the riding position on the VFR to be prohibitive. -Lucky me.
Point is: Bad shit can happen at low speeds.
There is a sliding scale between convenience and risk mitigation.
I tend to err on the side of caution.
Does it really take alot of time out of your day to throw some riding jeans and a mesh jacket on over your swim trunks? It's got to be better than the alternative.
Hope you stay safe, whichever way you decide to ride.