villagers stuck with professions

when I destroy the workstation a villager decided was their job while in a minecart, the villagers do not go back to default unemployed, or pick a different available profession. my biggest issue is with fishermen villagers. I have not traded with these villagers and I'd say there are about 40 stuck as fisherman, even though I swapped all 200 of my barrels for storage to chests and removed all other possible workstations from my house, which is 70+ blocks away from where the villagers are. The biggest issue is on jungle villagers, but original plains villagers are having issues as well