Cant join friends world with xbox live

When trying to join a friends world with xbox live it always says I need to authenticate to xbox live or disconnected from server, it only works when we're on the same WiFi connection.
Happens on my LG G Stylo (Android Marshmellow) and my Samsung Galaxy avant (Android Kitkat)