I've never invested in gold/silver for essentially that reason. We have this weird belief that gold/silver are "real" money. But... Why?
If we ever get to the point where the SHTF, the most valuable things to own will be barterable goods and firearms/ammunition. Not gold/silver. IMHO of course.
I think the psychology of the general populace in the case of a massive inflation event--or name your economic downfall--makes it really hard to predict what would happen, but it doesn't seem like anything good. In which case, what you mentioned, firearms and barterable goods, seem like the only immediate worthwhile commodities. In the long run, history does suggest that precious metals have good barterable value.
I've asked why, in another thread long ago, to which nobody satisfactorily gave me an answer (my point was you can't eat or drink gold, and you can't live in it, so how/why is it a doomsday currency?). Yet, it is the case in recorded history that gold is a reliable, tradable currency. So there's that, even if I don't understand it.
I've just never been comfortable with the companies who will convert your IRA's to gold, and so forth. Or start you a completely new investment account in gold. They play up the fact that gold will outperform the dollar over time, but there seems to be a baked-in assumption that the real value is that one day you can dump all that gold for loads of cash. Which assumes the long-term viability of the dollar. Which assumes that the economy did not go sideways. The only other meaning I can sus out is that if/when everything goes sideways, you have all this gold. But you don't really "have" it. Nobody is keeping $500k worth of gold in their house. Well, I guess somebody might, but most people won't have a place to put that, and would consider it highly vulnerable. I put my silver in a safe, but as I say, it's a fairly small amount and that safe, even packed full of gold, wouldn't be worth all that much. Maybe a bigger safe could get you a pretty large amount, not sure. Mine's fairly small.