Guardians and Elder Guardians "Ignore" the player

Sometimes whilst fighting the Guardians and Elder Guardians they will turn passive and totally ignore the player. They will just keep swimming into the wall or around and even if you're approaching and a few blocks from it, they remain passive.
This happens with both Guardians and Elder Guardians.
Elder Guardians sometimes even ignore you even if you hit it again and again, and they won't even swim away or use their spike attack.