When i join a friend's game, i cant Interact with stuff

When i join a friend's game, i Interact with stuff, but they wont count at all, to my friends im standing still, but to me im placing/breaking blocks, i cannot Interact with Doors, buttons, etc
My internet is stable, same goes for the host