Spectator players can enter other dimensions through portals

Spectator players can enter other dimensions through portals, which they aren't supposed to do.

How to Reproduce
1. Build a nether or end portal
2. Switch to spectator mode.
3. Go inside the portal.

Observed Result
You can enter their dimensions in spectator mode.

Expected Result
You shouldn't.