Loading friends world says "cannot connect to world"

While trying to connect to my friends's world it says connecting and then after about 15 seconds it says "cannot connect to world". My NAT type is open, his is moderate and we are trying to connect over xbox live. Once in the world I press the start button and invite him from there. Please help