Polls late in the season will usually reflect losses for P5 teams, with the occasional flip because of apparent SOS.
I think we see this after the CGs when for example #4 beats #8 in a tough game so #8 is now 10-3, but doesn't drop below a team that is 10-2. Obviously it takes a lot to overcome the extra loss when comparing an 11-2 team with a 10-3 team, but it happens. It's just not that common because clear differences in SOS are not that common.