The difference in charging from 110V via a 15 amp circuit and a 230V 30-50 amp circuit is significant.
Back of the napkin, 110V at 15 amps is about 1650 watts, or 1.65 KW. The max output you can get on 110V is 20 amp, but you'd need a special plug (and a 20 amp circuit breaker). So if you have a special 20 amp 110V plug, you could maybe squeeze 2.2KW from a regular 110V plug.
For 220V, you could get 6,600 watts at 30 amp, or 11,000 watts at 50 amp. So ~3x the power on the low side (6.6 vs 1.65 kw) and 5x on the high side.
These are general numbers, in real world the voltages vary from 110-120V usually, and 220-240V. But the watts will usually be about the same.
Level 2 charging is not "fast charging" by any means. I don't think there would be any harm in charging at 220V vs 110V, they are both relatively slow. The biggest difference is that most 110V circuits are wired with either #14 AWG or #12 AWG, whereas most 230V circuits would be wired with #10AWG or smaller (#10 AWG is good for 30 amps depending on the distance). The smaller the AWG gauge, the bigger the wire. So a bigger wire carrying less current will always have less resistance. Not sure there would be any big difference in terms of power loss due to resistance, but overtime it could add up.