Villager AI has them stand outside by night

I built a square village surrounding an iron golem farm. The buildings in the middle of each side have a door facing outwards leading onto a field. When night falls, instead of entering through the door, the farmer villager will stand outside, just three blocks away from that door. From the way he's standing there, he probably tries to interact with one of the villagers inside the house.