Villagers Despawning when entering Nether Portals

Im working on an iron farm right now in the beta so I have a villager breed linked up to a nether portal, but some of the babies when they go in are not there. I'm pretty sure it happens if you don't follow them through the portal right away. Whenever I try to AFK the farm, most of the villagers are not in there. It is really annoying because I've lost probably 15+ villagers from this. All my portals are linked up properly.