VIllager Trade UI immediately closes when a villager has iron, gold, emerald and diamond badges

Right-clicking a leveled up villagers (Trading/summon command) will automatically close the trade UI. This only happens when they is no POI (Jobsite/Workstation block) nearby. Sometimes the trade UI is displayed when right-clicking again.

How to reproduce

1. Spawn villager
2. Put a workstation block
3. trade with the villager once
4. Destroy the workstation block
5. Trade with the villager again