The span between 1940 and 1980 is not understood, nearly as I can discern, and is an obvious problem for the various models. The climate appears to have warmed 1920 to 1940 or so and then the warming trend abated. Then it kicked in again. The increase in CO2 levels in the atmosphere (which we can measure quite precisely) didn't show a corresponding decrease.
Then you throw in the enormous heat capacity of the oceans, changes in albedo, possible melting of permafrost, deforestation, changes in ocean currents, and a few other items and you have a very complex climate system.