Can't Destroy blocks with swords

At First I was riding my cool horse with a sword the I was trying to destroy random blocks in the village besides my house then I couldn't destroy any block with any sword even on a horse. cant destroy on a cart or a boat I can only attack mobs like pigs ECT.
But you can in survival