Some people can access my realm, but I can't.

After loading for a bit, the screen reports, "Unable to connect to world." With an OK button below it. My brother, on a different computer and different xbox live account, but the same LAN, had the same problem, but the other realm members have joined successfully. I am the owner.