I think Toyhaulers are more expensive cause they CAN be......

They're really popular/versatile.......
TT's are nice, cheaper, better layouts for more usable space and lighter (don't have the extra construction to have a removable wall....ie. Loading ramp).
I've thought about going that route also. If I was to get a side by side, this would be a definate option...
I hear you on the "hassle" of loading/unloading..........I have rented a motorhome a couple times and it's REALLY a pain considering you gotta get a ride to go and pick it up, bring it home, load it up and then bring it back home, unload everything, clean it real good and take it back............That's a pain, takes tons of extra camping/riding time to do.