I think some of the cost is disciplinary-dependent. I've never been to a humanities conference with a registration cost above $250, and there are usually pre-reg discounts. The idea of charging nearly $1000 for a conference to me seems risible. (I don't think Grace Hopper counts as academic, but rather as tech industry, where costs are even more ridiculous, but everyone's company pays so no one cares.) I think the other thing is that conferences don't necessarily assume that everyone is rich, but rather that everyone who does attend will have gotten assistance from their institution to do so. There's sometimes a large difference between accepted panelists and those who actually show up because of that. And some conferences will provide meals for attendees, more commonly when they're in a remote location. Ive been to several international conferences that did that to make attendance slightly more affordable. Which is to agree with your point that most academic conferences are pitched at an in-group, namely academics.

Okay, I have to run now, but I'll be back later with more!