Switching to off hand doesn't work

Switching items into the offhand doesn't work unless i open my inventory first and i have to manually switch it out instead of  using the key i have set for it.

Because the video file is too big, here it is as an unlisted yt video: https://youtu.be/k5Lh6pUClME