When spawning in a vanilla server, you will spawn just about anywhere within 100 blocks

When im using localhost to join my vanilla server, i keep spawning very far away from my custom /setworldspawn. I will spawn within 100 blocks. This is no were near where i set my world spawn. I am supposed to be spawning withen 20 blocks of the location.