Illager Patrol spawning without villages

Found my first Wandering Trader and almost immediately saw my first Illager Patrol nearby. I left the area to prepare to fight, but upon my return, the Illager Patrol had despawned. I searched the area, anticipating a nearby village, but found none within many hundred blocks.