Villagers disappear when someone goes to an pillager outpost

I was playing realms with my friend when he found a pillager outpost. When he came back I went to trade with the villagers but then suddenly 3 of my villagers disappeared. We tried to back it up but the villagers where still gone. Also my villagers were in a highly secured well lit room.