Getting kicked when removing someone else from the whitelist while enforce-whitelist is on

When I remove someone from the whitelist on my server while the enforce-whitelist setting is active I get kicked from the server with the message "You are not white-listed on this server". I just noticed this just happens when you're actually not on the whitelist but you are an operator so you can still join even though not being on the white-list.