Breaking some blocks in a village causes crashes.

One village in my realm has 20-30 villagers in it, much more than other villages in the realm but not an extreme number by any means. While trying to slightly move a patch of farmland I realized that breaking a composter caused all the villagers to instantly stop moving and within a few seconds I would be disconnected from the server. When I logged back in the last minute or so of game progress wouldn't be saved. I did this probably a dozen times between my Xbox Series S and my Switch with the same result every time. I experienced a similar issue a few days before when placing and breaking an ender chest, indicating to me that it's not just workstations that cause this. I haven't had this issue in any other villages on the realm.