Ridiculous amount of Phantoms after 1 night with no sleep

I have gone 1 night without sleep and about ~15 phatnoms are spawning. Progressively more spawn during the night and make it impossible to sleep. The only option is to wait out 20 mins for day.