https://en.m.wikipedia.org/wiki/Terry_stop
Well, that's not just "you're in a crime-ridden neighborhood so you surrender all rights"...
The United States Supreme Court held that where: (1) a police officer observes unusual conduct by a subject; (2) the subject's conduct leads the Officer reasonably to conclude that criminal activity may be afoot, and that the subject may be armed and presently dangerous; (3) the officer identifies himself as a policeman; (4) the officer makes reasonable inquiries; and (5) nothing in the initial stages of the encounter serves to dispel the officer's reasonable fear for safety, the officer may conduct a carefully limited search of the outer clothing of the subject in an attempt to discover weapons, and that such a search is a reasonable search under the Fourteenth Amendment, so that any weapons seized may properly be introduced in evidence.
I have a personal example I've talked about here before. Back in ~2003-04 when I rode a motorcycle, I had left work and was headed home, including a stop at the bank. In my haste, I parked the bike and was halfway into the door of the bank before I took my helmet off. I conducted my business, and as I was leaving the bank, already had my helmet on. So for any reasonable observer outside the bank, they never saw me without my helmet, and there could be reasonable suspicion that I had worn a helmet into a bank for the purposes of non-identification.
Two officers happened to be there and saw me come out of the bank. My conduct was unusual (1), a reasonable observer could conclude criminal activity may be afoot (2). They were uniformed so didn't need to verbally identify themselves (3), detained me briefly, and made reasonable inquiries as to my behavior (4), and ensured that no, the bank had not in fact been robbed. So based on #5 (the initial stages of the encounter where I explained myself and didn't give them reason to feel threatened), plus the fact that the bank hadn't actually called the police and reported a robbery, they did not attempt to search me.
This is a lot different then being hassled and searched because you happen to be a young black man in Manhattan wearing something other than a suit, which is my understanding of what most of stop & frisk was.