Yeah, that is a good point of course. We've all seen some team win 42-7 when they had a 4-0 turnover margin and two kick returns for TDs, but gained fewer yards etc.
I agree that it is not just score, I'll give you an example from two games that my team was involved in. This is the example I always use for this:
On October 28, 1995 I attended a game in Columbus in which the Buckeyes beat the Hawkeyes by 21 points, 56-35. It wasn't anywhere near that close. The Buckeyes absolutely dominated early and led, at one point by 56-0. Iowa did an admirable job against Ohio State's backups, waterboys, and cheerleaders thus making the final score the modest three TD win.
On September 23, 2006 I attended a game in Columbus in which the Buckeyes beat the Nittany Lions by 22 points, 28-6. It was MUCH closer than that. Ohio State and Penn State played a defensive slugfest for most of the game. Each team had two scoring drives prior to the last few minutes. The difference in the game was that Ohio State did better in the RZ. Ohio State's two scoring drives were both for TD's while Penn State's were both for FG's such that Ohio State led by eight, 14-6 late in the game. Note that this was still, at least possibly, a one score game. Late in the game Penn State was driving for a potentially tying TD when they threw an interception that was returned for a TD. The effectively ended the game by making it 21-6. Then Penn State stopped Ohio State and got the ball back and promptly threw ANOTHER pick-6 on the very next play leading to the final score.
I use those two games as my example because I think that really illustrates the point. Ohio State's 22 point win over PSU in 2006 was MUCH closer than Ohio State's 21 point win over Iowa in 1995.
One way that I try to think of it is by time rather than points. Ie, if I were a neutral* fan, at what point would I have considered the game to be "over".
*I say "neutral" because when my team is involved I stay interested long after the game is effectively over and I do this both ways. When Ohio State has a big lead I am the guy still worried that if the other team gets a 90 yard TD, then an onside kick, then another long TD pass . . . we might lose! Conversely, when Ohio State is down by a big score I am one of the last to give up, still calculating in my head long after everyone else has given up just how many hail mary TD's and onside kicks we need to catch up.