Connecting to other people via Xbox isn't working

Every time I try to connect to my friends on Xbox, I can't and it says unable to connect to world. But when my friends try to connect to me, they succeed. Are there any tips or tricks to make sure that you can connect every time you try?