Allays cannot be consistently interacted with while in minecarts

When attempting to give/take items from allays in minecarts, some can be interacted with, while others in the same conditions cannot. The attached video shows this in action, despite clicking on the allay with a cyan shulker box, I am not able to take it from it.