Wrong custom skin in multiplayer 1.16.0

I am in a world and I have someone join me but they are using a custom skin on their end and a custom skin on my end and it ends up making me use their skin but if I exit out it my skin it is when we both use a custom skin that it looks that we’re using each others skins instead of our skin. It will not happen if they are using a skin from the Minecraft store or the customizable skin. It needs to be custom skin for it to work.