Buried Treasure Map turns to regular map when regular map is in off-hand

I travel around with a large-sized map in my offhand to map out places I explore.
When I find a buried treasure map from a shipwreck, and try to right-click it in my main hand, it turns into a regular map and there is no "X" marks the spot present.
The item just says "Map" rather than "Buried Treasure Map"
This only occurs when I am holding a regular map in my off-hand.

I imagine when the game checks whether you are holding a map or a buried treasure map when you right-click the item, it assumes that a buried treasure map is a normal map because you have a normal map in your off-hand.