I was talking about that just last night at the academic conference I'm at right now ($855 just for registration, ouch -- there's a discount for students, but that doesn't apply to unaffiliated people who don't have an institution footing the bill). The people I was talking to who have organized conferences like this one in the past said that hotels are just incredibly expensive. (Charging $50 a head for a catered lunch that's just bread, cheese and cold cuts is one example.) I brought up the idea of holding conferences at universities (something fairly rare in my field). I was told universities are more expensive than I would think, especially WRT catering. I'm skeptical, but he's been on organizing committees and I haven't.

One thing that more academic conferences could do is to do what math conferences seem to do: charge a minimal registration fee and let people go out to restaurants for lunch and dinner (instead of only dinner, like at the conferences I go to). This relies on venues that are walking distance from restaurants, of course (and obviously, walking distance means different things for different people).