We had a "poll" here a while back and the middle point was that after about $5 million in net assets, a person could be considered "wealthy". It's a qualitative term. If someone says they are not, in their mind, they are not (unless they are lying of course). I might say, for example, we're all wealthy as compared with folks even in Europe, which is likely true for the most part. Someone else might opine it takes an extreme level of wealth just to avoid being homeless here, which to me, would seem ... obscure.
The "truly wealthy", e.g., billionaires and those near that level, can easily avoid income taxes if they choose, legally. My own definition of "wealthy" is "someone who doesn't need any income". That's intentionally glib.