Can't connect to switch worlds (Unable to connect to world) (Local network with 2 routers)

Whenever I try to join my friends switch game I get "Unable to connect to world" from windows 10 edition

But when I invite him to join my world he joins without any issues.He tried creating a new world on switch and invite me but it's still not working.

 Edit:

I found out what is causing the issue.

If Client 1 is connected to main router, while Client2 is connected to an new router connected in the main router, Client1 can't connect to Client2 since it can't reach the others routers IP.

But Client2 can join the others game since the new router is connected to the main router and can reach the IP.

 

When Client1 & Client2 was connected to the same wireless router and not the extender router it worked without any issues. (If client1 & client2 was both connected to the extended router it also works, so it's not a router issue)