Yes, I've gotten complaints as well. The GSA stock seating area is definitely more cramped than the GS, and at least with the stock seat, pretty thin. I'm surprised the Sargent didn't help things, as the passenger portion is considerably thicker than stock. I've never gotten complaints about the top box, but the tiedown hooks on the stock side bags can dig into the passenger's legs.
To be honest, I'd rather just ride it by myself. I'd like to have a better two up bike, but right now I can only go with one bike. The advice to go to the standard GS seating is a good one. Along with upgrading those seats to something better.
No idea if there is a way to still mount the top box with the GS seating though. Might have to get creative.