Blazes spawn without spawner

I was hunting for Wither Skeletons at my second Nether Fortress and the blazes got really annoying so I destroyed the spawners. after setting the difficulty to peaceful and back, more blazes spawned. so I switched to my x-ray texture pack and I couldn't find any more spawners. yet blazes continued to spawn and be annoying.