3 Villagers in panic state around Y:150 can't summon golem

So I tried to make a small iron farm with just three villagers and a zombie. I first made it in a creative world at the same height without a problem, it worked well. So I went to my own survival server and tried to make the same exact one, the same exact way. but it only spawned one or two golem then stopped. I can afk around it forever and not get one single ingot, but it works fine in the creative world in the same conditions.