Villagers have entity counts of over 50 each, extremely lags my game in area near villagers

Basically I have a few villagers trapped in carts near each other.
I get severe lag when I go a few blocks near the villagers.
F3 reveals that each have a high entity count
I'm playing on a Vanilla server and one other guy notices his villagers are starting to have high entity counts, he hasn't gotten lag, yet.
I have an AFK Fish farm near so maybe that could be attributing to the problem.
I just think zombies are tracking them and then despawn but for some reason the game thinks they are there still.

What I expected to happen was...:
Villager Entity should be 1 suppose.

What actually happened was...:
Depending on which I point at I get 50-109

Steps to Reproduce:
1. Put a villager in a minecart
2. Check F3 entity count
3. Maybe wait a few nights, kill mobs and you'll still see entity counts