Milk buckets restack after milking while standing on a ladder

Standing on top of a ladder with cow two blocks above me, I can milk it using a stack of buckets. The buckets fill up as expected and are put in new slots but as soon as I move or click on one of the buckets they all empty out and stack up again. You have to stand on top of the ladder, and not hang onto it.