Holding level 4 map or having in inventory cause lag.

I was lagging a lot in my world like if i had 10 chunks loaded , when i removed the map from my offhand the world stopped lagging a lot and when i removed it from my inventory i solved the problem. Render distanfe was 8 chunks, simulation distance was 4 chunks and there was no entity in the world and command blocks disabled. I can't believe maps makes the world so laggy, i basically have to ban them because i don't even wanna imagine if all the players explores a map at the same time.