Guardian Fish detecting me while invisible

i used an invisible potion to get down to the castle

What I expected to happen was...:
was not suppose to be detected by guardian fish didn't have any armor on and i didn't have anything in my hands

What actually happened was...:
the Guardian fish keep zapping me, have tried multiple times and the problem is still occurring

Steps to Reproduce:
1. Put a step by step guide on how to trigger the bug here
2. ... go invisible
3. ... go near near castle