When trading with Wandering Trader he does not take the emeralds

So I came across a wandering trader's llamas one night. All good. Found the invisible trader and opened the trade interface. 

If I trade holding the shift key, the trader gives me goods, but does not take the emeralds. If I trade without holding the shift key, he takes the emeralds.