Again, I think of this as being the entire Bay Area, and SF is only one bit of it.
SF is naturally small physically. I.e. if you look at this: https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population_density
San Francisco proper is the second most dense city to New York City, at about 2/3 of the density per square mile. But NYC houses 8M people while SF houses 800K because NYC is physically larger. So you have a completely different order of magnitude in scale between the two.
Where it gets REALLY interesting is if you filter and sort that list of the top 125 incorporated areas based upon metropolitan area.
The New York City Metropolitan area has another 56 incorporated cities on that list which house an additional 2.3M people.
The San Francisco Metropolitan area has another 5 incorporated cities on this list which house an additional 295K people. That's Daly City, San Pablo, East Palo Alto, Berkeley and Albany. Again, that's another order of magnitude. And the highest on that list is 49th in density. 23 of the NYC metro areas are above 49th on the list and are about 1.1M of the 2.3M from that NYC metro.
So yes, San Francisco is expensive, and relatively dense. Even if it reached NYC's density, the population would only go from about 800K to about 1.2M. So it would still be expensive. But if the entire Bay Area, including extending down the peninsula and across the Bay, was a little more like the NYC metro area, the ENTIRE Bay Area would still have high demand but the affordability point would be much more accessible than it is today.
I think this cuts to the crux of the issue.
This started with the issue that SF artificially lowered its density from being in competition for the most dense major city to just being the second-most dense by a healthy margin.
But in truth, being that dense hasn't been a slave. So now we're moving out, saying "If only San Jose or Oakland were more dense, THAT'S the magic bullet," as if being like NYC isn't both a rarity and a massive undertaking/accident of history. And our comparison point is ... the NYC metro, where affordable housing is also a nightmare.
I'm not saying you're wrong that more housing should in theory lower that price point. You're right. But when it all works out, density on its own doesn't seem to correlate with affordable housing, and it's worth keeping that in mind when we want start imagining semi-unfeasible things like getting Santa Rosa to be more like Queens. It's all a great theory, but the same kind of market theory holds that forces shouldn't produce real estate prices like this to begin with.
(I also wonder about what I'd call untapped demand. I remember reading The Power Broker, in part about the development of NYC. And they ran into an issue of early traffic problems. The answer, much like our density think, is to build more roads, bridges and tunnels. And it turned out that every new pathway they built gave more folks the bright idea to drive, so each bridge in turn was overwhelmed and we have the joy of modern NYC traffic. Whatever bubble is out there isn't deterred by paying $1,300 per square foot)
