Villagers Not Losing Their Professions

I made a villager trading hall and placed a Lectern in front of one. He took the profession, but I didn't like his trade, so I broke the lectern. He didn't lose his profession, so I thought that it was not the right time of day. But, I camped out next to him for an entire Minecraft day, and he never lost the profession. This happened twice after that with a Fisherman and a Cleric villager. I never traded with any of the villagers, so it makes no sense why they wouldn't lose their professions. I searched online, and I see that this has been a problem before. I don't see any solution to the problem though.