Online Play with Friends not working

Every time I try to join a Friends world, it says "Unable to connect to world." We're on the same WiFi, have a stable connection, on our Xbox Live accounts, and are Friends on said accounts.