I think we'd all agree that at some point a city becomes to large to be a real "college town".  My guess is that limit is something like 200,000 metro area.  Above that, the university influence is too diluted, though the area around said U probably will seem collegy.  It could still be fine for students, it's just that five miles away you wouldn't have any/much evidence there is a large U nearby.
If you can walk through downtown of a city and see little evidence there is a large U nearby, it's not, to me, a college town.  It's a city that has a U in it, like Cincinnati.