I have taken both the Alaska and BC ferries a few times and have really enjoyed them each time, so I try to work them into my adventures. On all my trips though, I have either had a room on the long hauls, or just wandered around without setting up "camp" on the short ones. Here are a few thoughts:
If your schedule allows, consider breaking the trip once or twice for layovers. I do not think it has a big effect cost-wise versus the full Bellingham to Haines long haul. There is enough to do in Juneau or Sitka, and even Ketchikan and Prince Rupert, to make it worthwhile. If you do this, some of the legs are so short that worrying about sleep is moot. Juneau to Haines is only about 6 hours, for example. On the other hand, on a longer leg, you could pay for a room if only for the change.
People do camp on deck, and there is a real "culture" there. I will defer to a camper's response rather than speculate. The larger ferries at least, have a covered deck with ceiling mounted heaters, so you are not totally exposed to the elements. One thing that is obvious is that you need a buddy so you do not feel you need to guard your stuff 24/7.
You will encounter crazy scheduling at times. I had a 3:30AM departure from Juneau to Sitka once that might make squeezing onto an already "claimed" camping deck iffy. On the other extreme, I had an 11PM arrival in Prince Rupert this summer, making getting settled into land-based lodging something you really need to pre-plan, particularly if it is raining and cold.
The ferries have self-service laundries and pay-per-use showers, so you can get your gear (and yourself) all sorted. There are short periods every few hours when they allow you to "visit" the vehicle deck and fiddle with gear (this is mainly for people with pets in their RV's).
Broadly speaking, the scenary gets better as you go north. In/out of Sitka, Juneau to Haines, should be done in the daylight.
You can arrange everything on-line or by phone. Everyone I have worked with has been real helpful.