Blaze aggro - undesired pack aggro effect

I have made a blaze farm using a spawner and when player A stands inside it and the farm is running its fine. When Player B moves around the surrounding fortress and aggroes a blaze from a far the blaze from well inside the farm starts shooting at the player B's general direction even though he is 10-15 solid blocks away.