Nether Portal Destination Glitch

When I was in the nether, I built a nether portal at the nether fortress so that I can access it easily from the overworld. When I enter the nether portal from the nether fortress, everything seems to be alright. However when I enter back into the nether from the portal I enter at the nether fortress, I expected myself to exit to the nether fortress nether portal I built, but it exit me from my other nether portal instead...