Iron Golems will become aggressive towards the player immediately after spawning

The bug affects 1.20.1, I am not sure if it affects 1.21.3 or other versions.
I have a desert-only hardcore world where my base is in a desert (obviously). I also have a villager breeder with ~15 villagers inside, and because there is a cave nearby which spawns zombies, the villagers will spawn an iron golem every possible chance (every 500 game ticks). I kill these iron golems for their iron. However, sometimes when I kill an iron golem, the next one that spawns is immediately aggressive without being attacked. Leaving the world and re-joining before the iron golem sees me appears to solve the issue.