Allay causing entity crash in woodland mansion

I arrived in my world in front of a Minecraft mansion and the game crashed.  According to the crash report the entity that caused the crash was the Allay I assume was in the mansion (pillager towers don't spawn in dark oak) and caused the game to crash multiple times. I am now unable to go in my world at all without this crash happening again again. I am also unable to turn to peaceful my difficulty is locked on hard I don't have cheats therefore I can't kill all entities. Please fix!