"Setting Up Custom Skins" Non-Responsive

While playing the game would go into "Not-Responding" for around 30 seconds whenever in the console it showed "Setting up Custom Skins" despite people not actually joining or leaving the game, as if this was running on a timer.

This does not happen every time I play, seems to just happen on a luck of the draw basis.