It is not simply the number of aliens. It must be some part of the AI procedure on some condition that gets too slow. I had moments where the hidden movement seemed to have freezed computer... taking like 4 minutes. Then on next turn without any alien being killed, suddenly the alien turn then passes in 8 seconds. Then other turn.. back to 4 minutes.
Most of time it seems to happend when the aliens loose sight of my troops.