Unable to Join Realm after accepting invite

I am able to receive a friends invite and it shows he is on it in xbox live but does not show in the minecraft client. When I accept the invite I get the failed attempting to join realm text.