Part Visibility component breaks when in multiplayer

My world has a custom block that uses part visibility component and I am the host, when another player joins my world the players sees the custom block like the part visibility wasn't there, but to the host it looks fine.

In the picture on the left side is the host, the custom block pillar's middle is smooth and on the right side is the player who just joined, the player sees a clearly visible bunch of pillar stack on top of each other.