If the average district size varies by state, it matters. It's not intended to, but in practice our states can't be divided by an exact number of voters per district, because: math.
My recollection from looking at this is that the less populous states gain a slight advantage from this, but I will freely admit that my recollection has often been faulty.