Crops don't grow outside a certain distance to the player, which is smaller than simulation distance

What happens

My render and simulation distance is set to 32 chunks, but when I get far away from the crops they stop growing despite being in render distance and start growing again when I go closer

What I expect

Simulation distance to determine the distance that the crops stop growing