Can't set/unset banner markers on maps with 'vanilla' datapack disabled

Go into a world, grab a map and a banner.
Try put it on the map. Hey it works!
Now disable vanilla with /datapack disable vanilla
Try again. It won't work...

I expected there might be a recipe for it or some sort of action detection that is in the vanilla pack. But there wasn't so I was confused.