I think we'd all agree that at some point a city becomes to large to be a real "college town". My guess is that limit is something like 200,000 metro area. Above that, the university influence is too diluted, though the area around said U probably will seem collegy. It could still be fine for students, it's just that five miles away you wouldn't have any/much evidence there is a large U nearby.
If you can walk through downtown of a city and see little evidence there is a large U nearby, it's not, to me, a college town. It's a city that has a U in it, like Cincinnati.