When exiting a container or villager UI, the player cant swing for a second

when trying to get a villager I noticed a slight delay from when I exited the UI, and when I could swing my axe to break the lectern. I dont know if this is intentional but it drastically slows things down.