banner pillagers in raids giving bad omen

there are no villagers alive at my base ( the mob despawning bug that 1.14.2 fixes ), but there still is a village where raids can start, and multiple raids at once creating a hell storm where the best gear is getting destroyed in 3 seconds. this is on a java realms server. thank you