Villagers disappearing out of farms.

I have farms that the villagers are set up in to work in and the farming one always disappears after a while of the chunks being unloaded. I'm always having to replace the brown coat and it's getting very frustrating.