Iron golems attack when changing difficulty

I made a couple of mob farms near my base recently and i went to check to see if my villager trading hall was okay. Went inside and instantly got ganged up on by five iron golems. I killed the golems and they respawn and keep attacking. I let the golems kill me, Still didn't work. I Didn't even attack any of my villagers.