Getting too close to a trapped Illager crashes realm and causes a rollback

In our realm we have a large man made tree with a large flat platform on top, and there are pillagers that have spawned up there, I previously went up there and saw them, aggravated one but then jumped down. But the following day, whenever any of us go up to the top of the tree, the realm becomes unresponsive and eventually crashes, kicks us from the game and rolls back to presumably the last backup. I think this has something to do with the pillagers since it only happens when we go up there, and happens without fail.