Endermen are always angry

I'm playing both in hardcore mode and survival, and everytime I see a Enderman it's always angry with someone/something.. even if I don't look at them in their face/eyes...

Fortunatly it doesn't attack the player.. But it still emit scary sounds..