Use milk to cure mining fatigue, guardian curses me immediately

Attempting to clear ocean monument. Using milk to clear mining fatigue, however as soon as I’ve cleared it, guardians curse me within 1 second. I have cleared ocean monuments before so this isn’t supposed to happen that fast. I can’t break one block because I’m constantly fatigued.