D1 is going to keep growing. Some schools will drop (eg. Savannah State most recently) but many more will replace them (recently / next are Dixie State, UC San Diego, Merrimack, Tarleton State, North Alabama, Bellarmine....) I don't begrudge them the option..... To be fair, it has worked out well for some schools (Boise State and UCF were in D2 until 30-40 years ago, North Dakota State has dominated FCS football and been decent in basketball since joining D1).
It won't happen for basketball, but I do think it would make some sense to split D1 for sports that have a lot of programs (softball, soccer, volleyball, etc.), but it'll never happen.
As for the Basketball tournament, there's no reason not to go back to 64 teams. That's how it is on the women's side with the same number of conferences, and the First Four teams are rarely very good (VCU's final four run aside)..... They won't ever shrink the field further for the same reasons they wouldn't go below 64 in soccer, baseball, softball, and volleyball. Every conference is going to get an autobid while still making room for the top 30-40 teams overall.