When playing Realms, I almost always have default skin

I own a Realms server, and when I log in, more than half the time my skin looks like Steve to me, though other players see my custom skin. Sometimes I can fix this if I quit out of Minecraft and restart, but often that does not fix it at all. This ONLY happens on my Realms server.