Minecart exiting teleport

When I exit mine cart I'm teleported through a wall into my castle which is 1 block thick and the minecart is right next to the wall.Its highly annoying as you have to go around the outside to get rid of your minecart from the track