Unable to punch mobs with anything except with swords

I switched from creative to survival and then to creative again. I first tried punching a mob with a andesite block then it didn't got hurt. I tried with other blocks it didn't work. Then I tried with a sword and it worked.