Invisible force attacks me

I was walking around my little village at night placing torches, when I started taking damage. I wasn't able to attack it, and it sounded like there was an enderman there, but I never aggravated one. I was able to go in my house without it coming in.