Even Gordon doesn't have the best reputation in the NFL, due to low career ypc numbers.
The Chargers OL has also been a shitshow basically the entire time he's been there.
Aside from the reports of Brady wanting to be close to NYC area where his family has moved to, I'd guess a big part of him picking Tampa over the Chargers was the OL situation. Tampa had the #7 OL unit in the NFL according to PFF's rankings. Chargers? 29th. Aside from that, I'd also argue that Tampa's WR weapons are more physically talented/gifted than the Chargers'. Mike Evans is superior athletically to either Keenan Allen or Mike Williams, and Chris Godwin is superior athletically to Keenan Allen for sure. And as far as TE's go- he doesn't have the insane production yet to match his insane talents, but you will not find a better athlete at TE in the entire league than OJ Howard. Simply doesn't exist.