Spectating through portals then switching gamemodes will teleport player

In 1.21.4, all versions after, and potentially some versions prior, when the gamemode is set to Spectator and the player collides with a nether portal or an end portal (where they would be teleported immediately if they were in Creative mode), switching to Creative mode through F3+F4 at any moment after colliding with a portal will cause the player to be teleported to the destination, even if the player isn't in a portal when they switch their gamemode from Spectator to Creative.
The expected behavior after switching to Creative mode is to not teleport to a portal, unless the player is explicitly in a portal when they switch.
This can be reproduced with either a nether portal or an end portal; first place the nether or end portal, switch the gamemode to Spectator, fly through the portal, and switch the gamemode back to Creative while not in the portal. The player will be teleported even if they aren't in the portal.