Pillagers attacking player in spectator/creative after switching gamemodes from survival

Pillagers will attack me despite me switching gamemode from survival to spectator/creative. but no other mobs appear to do this, and they'll follow me and etc as if I was in survival.