Villagers outside of fence

Usually when I try to prevent Villagers to wander off, I use fences to surround them in an open area. However, sometimes when I join the server, a villager for some reason is outside the fence, even though I did nothing to make that happen.