You have the right to be selfish and irresponsible.
But it doesn't mean you have to be.
You have the right not to wear a mask around other people.
But maybe you should.
There's a degree that I, as a libertarian, believe in the right to be selfish and irresponsible. I think that seat belt laws for cars and helmet laws for motorcycles are an infringement on our rights.
But you wouldn't catch me in a car without a seat belt or on a motorcycle without a helmet. I have the right to be stupid; I also choose not to exercise that right.
I'm not sure that I agree that you have the right to not wear a mask around other people in the middle of a pandemic, though. For a very simple reason: masks are only partly about self-protection. They're even more about protecting others FROM you.
In my city, you can own a gun. In my city, you can fire a gun (at a human or animal) if you believe they are a legitimate threat to your person or other persons. However, you are NOT legally allowed to fire a gun within city limits otherwise. I can't go into my backyard and set up some targets and just start plinking. Because the population density here is so high that even if I had no intent to hurt someone, the chances of it happening are too high if we allow people to be popping off guns within the city. Where my BIL lives in Oregon, he can go out into his pasture behind his house and start plinking, because the conditions are different there.
A mask mandate in the middle of a pandemic is a much smaller example of the same thing. People think they don't have COVID just the same way they think that if they're plinking their gun in their backyard they're responsible enough not to hurt anyone. But there's a negative externality there that if a BUNCH of people in a city are plinking at targets in their backyard, eventually someone's gonna get hurt, and if you have large-scale non-masking in a heavily populated area, you're going to spread the virus far wider than if you didn't--even if nobody who spread it knew they had it.
So we have mask mandates that say that if you're in a situation indoors, or if you're outdoors in an area where you can't responsibly distance from others, you have to wear a mask. Given the pandemic, I don't see this as a rights issue.