I can relate to that bwar. From my pre-Bloomington years, I have a cluster of friendships that Vegas would give long odds to end before death, including one from kindergarten through the end of college. But at IU, with rare exceptions, I never really added any that didn't belong to my wife. She's since moved to Buffalo because she got her dream professorship, and I still hang out with them, even threw a party recently and will do it again soon. But whereas that's all quite nice and we're authentic around each other, there are rarely any shenanigans. We just sit or stand, drink or don't, and talk - current events or science/philosophy. Pleasant, even fascinating, but not brotherhood. It's not close to the same.
I guess that means all I can recommend is stay close to your wife (which I think you are), keep a pet (which I think you do), and find as much pleasure as you can in your moments alone. Not because you should pack up and enter your life's cat lady zone, but because while it's safe to bet more friends - even great ones - can/will enter your space, that's not necessarily in your control. And if you believe in the principle that happiness is always a choice, but not always easily made, these other things will help you choose it more easily no matter the friends that move away or how slow you meet the next ones.