Iron golems attack for no reason

For some reason, iron golems attack me in survival, even though I never killed any villagers or cats. I never smacked a villager. We moved them, but that shouldn't trigger them attacking. I've been killed many times and its getting annoying. They even attack in peaceful mode. Also, we did send a few villagers to get killed by cactus, but that shouldn't make the golems attack the players. To recreate this, make a trading hall that can spawn iron golems, and just move the villagers around. For some reason the golems attack the player. Even in peaceful mode