One, maybe two, of the top 12 will end up 8-4 or worse. Three to 6 of the top 20 will end up 7-5 or worse. By the time the committee starts paying attention, a lot of this will be evidence, and their rankings won't show any lingering bias from early polls. If you beat up a #5 team early that is 3-3 by the time the committee starts looking, it simply won't matter to them whether they started ranked 5th or not.
Back when polls were part of the BCS, it matter some of course because the polls won't shift around as quickly as they might.