Items that can be off-handed can duplicate when switching it out

Once, while in elytra flight, I switched out my Totem of Undying with a map in my offhand. When I closed my inventory, I realized that I was still holding the Totem in my off-hand but when I checked my inventory, I had another totem but the map was gone. This happened in 1.14.60 but when the game updated, the totem had still not changed back to the map and was able to use the totem as a totem. In 1.16, this bug happened again.

Steps to Reproduce:

• Switch out an off-handable item with another off-handable item

Observed Results:

The item that was originally in the off-hand would be switched with another off-handable item. Upon closing your inventory, the originally off-handed item would take the place of the switched item and still be in your inventory.

Expected Results:

When two off-handable items are switched, they should swap places in the inventory. Not duplicate.