Honey doesn't work against withering effect

Usually honey can remove all potion effects, weirdly enough honey works against the withering effect of normal wither skeletons, but not the wither itself, while milk still works for both.