Moving through portals in Spectator mode and changing game mode after moving away teleports the player through

Moving through a nether portal, end portal or end gateway in Spectator mode and switching to a different game mode after moving out of the block acts as if the player is still inside the block and teleports them.

How to reproduce:

1. Execute one of the following commands:

   /setblock ~ ~ ~2 minecraft:nether_portal

   /setblock ~ ~ ~2 minecraft:end_portal

   /setblock ~ ~ ~2 minecraft:end_gateway{exit_portal:[I;0,0,0]}

2. Switch to Spectator mode.
3. Fly through the block you placed.
4. Switch to Adventure, Creative or Survival mode.
   → You are teleported as if you were still inside the portal/gateway.

Expected result:

The player would not be teleported through a portal or gateway when not inside it.

Observed result:

The player is teleported through a portal or gateway when not inside it by moving through it in Spectator mode and switching to another game mode.