Major camera delay when looking down

When trying to  look down the camera has major lag and also sometimes does not want to look down at all.   

 All other  directions work fine.  

It usually happens when mining or walking.