Allays won't pick up items from raids unless dropped by a player

I just finished building the sorter to my raid farm and just realized that my allay will not pick up totems dropped from the raid. It will only pick it up if a player drops it.