Villagers are not working at their stations.

I have 3 villagers trapped in a 1 block area that are meant to be scared by a zombie when they see it and when the zombie leaves their line of sight they either sleep or work depending on the time of day. I know this issue has nothing to do with line of sight with the zombie because the villagers are able to successfully sleep, however when it turns to day and they lose sight of the zombie they don't work, they just stop. This doesn't happen all the time since my farm has worked before, however sometimes it happens and I have to kill and re spawn villagers since once they do this they never work again.