Village popularity doesn't increase

In a word with a village that I have a low popularity, I have bought more than a full stack of bottle o' enchanting from clerics without attacking them anymore, and the iron golems still attack me, even if they were newly-spawned. As far as I can tell, my popularity is somehow not increasing at all.