Hoppers take items out of chest minecarts slower when there are multiple minecarts in the same place

How to reproduce

1. Place the hoppers and minecarts as shown in the screenshot.
2. Put multiple minecarts with a chest on one of the hopper, put only one on the other.
3. Put a stack of items in one minecart chest on both sides.
   → The hopper with multiple (empty) minecart with chests above it will take longer to take it all out.

Reason

See MC-124099