Well, take two hypotheticals, Team A goes 0-2 out of the gate, and finished 12-2, Team B goes 12-0 and finishes 12-2. Similar schedules and opponents.
Team B is probably ranked #1 at 12-0, drops to maybe 5 after loss one and drops to maybe 7 after loss 2.
Team a is ranked say 10 preseason and drops out, but climbs back in and is ranked say 9th at 10-2. Yeah, they end up higher.
I still viewed it as fun, given no system is "perfect" (e.g., aligned with what I could think).