Hat dissapears on some servers on join

When joining a server, the hat disappears, making some skins look awful, to get your hat back, you need to click ESC and wait a few moments until you see your hat again. This is happening on most servers, I don't like it.