Pillager Captain in Patrols despawning

Multiple times during my survival world, and once in my brother's world, I find patrols, I then go to my base (usually 10-24 blocks away) to get items to fight them, I come back from the base and see the Captain has despawned and only the non-captain Pillagers remain. I had my render distance to 10 and simulation distance to 6. This shouldn't happen.