villagers remove fence gate

I have an infinite villager breeder with a gap for baby villagers to walk to and get driven away with the water flow. for ease of access, I had a fence gate above the gap to enter and leave as I please. After some time, the fence gate is removed and the breeding villagers have mixed in with the rest of the villagers. This has happened twice