Iron Golems stop spawning

Iron Golem farm stops spawing Golems and cats after you leave the rendered area and come back.  Only way to fix is replace all villagers which is not an option to keep doing in survival.