Villagers despawns when I go to other dimensions

Villagers are despawning when you go to other dimensions like nether or the end. They keep despawning by one everytime I go to nether. This is happening when there are 20-40 villagers in a place.