Cannot connect to friend's World without using a VPN

I cannot connect to my brothers server even on the same WIFI. We can only connect if we both are connected to a VPN. Even if one of us switches the VPN off then we get disconnect from the server and cannot connect again. It keeps on showing unable to locate world.