Unable to connect to world

I cannot connect to my husband's world. I can have friends join my world and my husband can join my world. Friends can join his world but I can't. We have thought of many possibilities of why I cannot join his world but none of them make any sense since he has no problems joining mine. We play on the PS4 and for whatever reason I cannot play on his world. Please help!!!