Guardians attack players with invisibility

What did I do?
I took an invisibility potion whilst working on my guardian farm. I took off all my armour and only held sand in my right hand.

What happened?
Guardians targeted me and I took damage.

What did I expect to happen?
I expected the guardians to ignore me like they did in prior versions.