Xbox Controllers buggy

I am using an iPhone 11 with a wireless Xbox one controller and sometimes it makes my characters vision drift around on its own, especially when I barely move the RS joystick jus a little bit, it starts moving on its own. Sometimes my character starts moving on its own and It takes a while to make it stop moving  it also happens on my iPad 6 gen