Endermen will occasionally aggro on the player even if he/she has done nothing to make them aggro

On the realm that my friends and I play, we have noticed on many occasions that sometimes endermen will attack unprovoked. We’re not sure what’s causing it, and theorize that it could possibly be endermen taking some form of damage and becoming aggro, as the last time it had happened an enderman had fallen into the void in the end region and teleported back to an island aggro, then subsequently attacked me and knocked me into the void