Endermen go back to neutral after being hit by player

Pretty much what the summary says. Also, this only happens if you look at the enderman's eyes, not just simply hitting the enderman

How to reproduce:

1. Get an enderman angry by looking at him
2. Hit the enderman
3. Notice he goes back to neutral and no longer attacks you