Villager unable to change profession bug

There is a annoying bug with the villagers

when you have 2 nitwits and give them a profession they become it, but when you do the blocks for that particular profession away they do not change back into nitwits.
I tried to let them change then again into a new profession by placing a new profession block(example lecturn) but they stay stuck in there old profession.