Allays are affected by sword damage.

I have allay that are mine when giving them an item but when accidentally hit with a sword they suffer damage and die.
The bows, crossbow spikes when attacking the allay these are not affected by the damage only with the sword.