Villagers stop harvesting crops after a while

Villagers stop harvesting crops after a while. They can plant seeds, but wont do anything else than walking in the field. The villagers often group up innside their houses at day time.