Guardians Not Re-spawning Once I approach my Farm

I created a guardian farm on my realm world where I drop the guardians down a lava tube onto some hoppers and then their drops go into a chest. I have a structure underneath the area where they spawn where the end of the tubes, hoppers and chest are, so I can walk around and collect the drops from the chest. When I first log on, the guardians spawn pretty good (usually only in about 6-8 of the 25 spots - but that is probably because of where I am standing). Once I go and check the chest and then leave the area (and return to where I was previously standing), the guardians will no longer spawn (regardless of how far or near I am). I have to log off, wait a little while and then log back on before they start to spawn again. I have let the farm run for up to 4 hours and it works great as long as I don't check the chests (or walk under the spawn points).