Endermen not being aggressive and despawning immediately

When I tried farming enderpearls in the end with fire aspect, endermen would teleport away, become passive again, and despawn. Only if I was lucky would one still be around for me to kill.