Shovel Efficiency IV conflict

I created a shovel with Efficiency IV and Unbreaking III. It digs incredibly fast, possibly too fast. Often when digging dirt or sand if I hold Left click and dig out an area, there are several areas that I am unable to walk through. It's as if there's an invisible block in my way. If I place a block into this invisible blocks place, the original block (dirt, sand, gravel) reappears. I am able to redig it. Only after that can I travel through the empty space unhindered.

I have tested this with a shovel with Efficiency II and did not have any issues.

It has something to do with the Efficiency IV enchant.