Xbox one controller Mining/Attacking

When I look try to attack a fellow player or mobs or mine a block I look up and I cant look back down at all so plz try to fix it.It happens on xbox one controllers only so far.