Villagers gets stuck climbing on ladders

Villagers will attempt to climb ladders but get stuck on hatches. If I open/destroy the hatches they will climb the ladders but get stuck on that floor of the building. The only way to get the villagers down is to destroy the ladders (If I don't, they will just climb the ladders again and get stuck.)