Too many villagers!

I created a village with 36 beds and 32 work stations.  The village was stable for a while, however now, I keep on getting more and more villagers.  I have over 12 nitwits in a small area that I have been able to count. (they were all swimming in a little area, not to mention the other random ones throughout the village. It is practically impossible to count them all and I think I may have more Golems than villagers at this point!