Taking damage when planting

When holding the use item/place block button while planting seeds and at the same time stepping from farmland block to dirt block I take damage. But If holding the use button while looking at nothing I don't take damage.