Villagers unclaiming work site randomly and or rapidly

Villagers spawned will randomly abandon their workstation, even with no new workstation existing, and will soon after reclaim it.

Steps to recreate:

1. Build enclosure as in first picture
2. Place down 10 beds and 20 brewing stands, as in second picture
3. Spawn 20 adult villagers
4. Wait until night time, then go to sleep
5. During the day, you'll notice the bug occuring.
6. To test this more accurately, after step 3 place a repeating command block with the command "effect give @e[type=villager,nbt=!{VillagerData: \{profession: "minecraft:cleric"}}] glowing 99999" (without double quotes) and power it. Villagers who lose their pofession for even 1 tick witll start glowing.