Based on what I've noticed, a successful athletic (in particular football) program has to start with the school's overall leadership.
For example, UW had Donna Shalala and she hired Pat Richter to lead the AD office. He then hired Alvarez. Her successors were also highly supportive, and still are. Another example is that Stanford's rise happened under John Hennessy, who was also very supportive of football. He hired Bob Bowlsby, who hired Harbaugh and then Shaw (Bowlsby also hired Kaptain Kirk while at Iowa).
This support is a huge part of the deal.
So, what sparks leadership having an interest in athletics? It just happens? Big Donors push for it?
Programs like UNC just seem to hire and fire with no discernible plan. The team might win 9-10 games one year and then fall into mediocrity or worse and then the fire and hire again. Maybe they need to fire and hire at a higher level. Maybe it needs Big Donors, who may be lacking at Illinois.
Those two programs SHOULD be able to field decent teams consistently, one would think. They are flagship universities in larger states. UNC does have to compete with other in state programs like NCSU to a greater degree.
I think the Purdue experience is somewhat instructive here. Purdue had a gigantic missed opportunity with Tiller. He revitalized the program. He made us relevant again. And the Athletic Department all the way up to the school president took it for granted, talking about how they didn't want to participate in the "arm's race" of college athletics. Morgan Burke liked taking the cheap way out, and the folks in power above him enjoyed that too, because I believe they thought that successful athletics was a distraction to the university, not a cause for celebration.
Purdue was one of those schools where not only did we NOT fund any portion of our athletic budgets out of tuitions or student fees, but we actively required the athletic department pay the university a stipend each year for the honor of being associated with the "Purdue" name. I believe that we have ended the payment from the athletic department, but I believe we still operate entirely as a self-funded entity and don't take any money from tuition or student fees.
What happened? Tiller got fed up with the administration that wouldn't support him. Our facilities were at the bottom of the Big Ten. We didn't reinvest or even attempt to make it better. We went the bargain-basement hire for Danny Hope, we then pivoted and paid minimally competitive salary for Darrell Hazell, and all the while basically tried to short-change the football program in any way possible--which I'm sure was seen and recognized on the recruiting trail.
As much as I think Hazell was a bad coach, I simultaneously recognize that he was largely put into a no-win situation, given that the university didn't support the program.
That's all changed. Morgan Burke was replaced by Mike Bobinski. Mitch Daniels is our university president, and as a politician he understands the value of perception and recognizes that athletic success excites and energizes the students and fans, and helps to keep the "Purdue" name relevant.
So we got Brohm, and we've kept him not only due to giving him a highly competitive salary, I believe that he stuck around instead of going to Louisville because he believes deep down that the athletic department has his back, and will give him what he needs to be successful. Maybe that's never CFP, because we're still "just Purdue", but give him what he needs to make trips to the B1GCCG in December more than just a pipe dream.
You have to have an AD, a President, a Board of Trustees, all on board with athletic success. Because it's a competitive race on the recruiting trail, and kids aren't going to go to a school where they believe their coaches are led by people who penny-wise and pound-foolish.
