Taking random damage while building

I was nerd poling up into the sky with cobblestone, and took about 3 hearts of damage randomly. I thought it was just a one time thing, but then a couple of blocks later, my health completely drained and popped my totem of undying. This is not the first time this has happened.