Allays teleporting through a nether portal shortly after going through

Whenever I bring the Allays to a certain location via a Nether highway, for some odd reason, they will always snap back to the portal and teleport though it about 10-15 seconds after coming through. For some weird reason, it ALWAYS happens at a specific spot. I brought them through other nether portals before to different areas, and I never had trouble then. The place where I have trouble is the only one with negative X coordinates. I tried breaking the portal right after going through. After 10 seconds, they simply vanished into thin air, presumably into the Nether. I'm currently looking for them right now.