Village Reputation is not going up/Iron Golem aggro

Put the summary of the bug you're having here

What I expected to happen was...:
I expected that my reputation would eventually become okay after trading and completing a raid.

What actually happened was...:
My reputation is just worsening with villagers and every Golem aggros to me randomly.

Steps to Reproduce:
1. Punch a villager a couple of times trying to get them out of a boat
2. Let gossip spread of how bad you are
3. Have Iron Golems aggro on you

 

attached is the number of golems I have lead out of the village.