[MC-263663] Extreme world saving performance regression

Description:

Since 1.18.2 (not 1.18.1 or earlier) and up to the current version (1.20.1), saving a small world takes several minutes on a dedicated server.

Specifically, on 1.18.1 I can create a new world and immediately stop the server. The save only takes about 8 seconds. Starting and stopping the server again takes only 2 seconds to save.

1.18.1 Save after world creation

[19:08:54] [Server thread/INFO]: Stopping the server
[19:08:54] [Server thread/INFO]: Stopping server
[19:08:54] [Server thread/INFO]: Saving players
[19:08:54] [Server thread/INFO]: Saving worlds
[19:08:54] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:overworld
[19:09:02] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_nether
[19:09:02] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_end
[19:09:02] [Server thread/INFO]: ThreadedAnvilChunkStorage (world): All chunks are saved
[19:09:02] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM-1): All chunks are saved
[19:09:02] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM1): All chunks are saved
[19:09:02] [Server thread/INFO]: ThreadedAnvilChunkStorage: All dimensions are saved

However, with the first 1.18.2 snapshot, 22w03a, creating and saving a new world takes ~2 minutes and 10 seconds, and starting and stopping takes about 1 minute and 30 seconds. This is an incredibly a long time for what is a very tiny world (12 MB).

1.18.2 Save after world creation

[19:13:01] [Server thread/INFO]: Stopping the server
[19:13:01] [Server thread/INFO]: Stopping server
[19:13:01] [Server thread/INFO]: Saving players
[19:13:01] [Server thread/INFO]: Saving worlds
[19:13:02] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:overworld
[19:15:11] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_nether
[19:15:11] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_end
[19:15:11] [Server thread/INFO]: ThreadedAnvilChunkStorage (world): All chunks are saved
[19:15:11] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM-1): All chunks are saved
[19:15:11] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM1): All chunks are saved
[19:15:11] [Server thread/INFO]: ThreadedAnvilChunkStorage: All dimensions are saved

1.20.1 Save after world creation

[19:20:10] [Server thread/INFO]: Stopping the server
[19:20:10] [Server thread/INFO]: Stopping server
[19:20:10] [Server thread/INFO]: Saving players
[19:20:10] [Server thread/INFO]: Saving worlds
[19:20:11] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:overworld
[19:22:15] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_end
[19:22:15] [Server thread/INFO]: Saving chunks for level 'ServerLevel[world]'/minecraft:the_nether
[19:22:15] [Server thread/INFO]: ThreadedAnvilChunkStorage (world): All chunks are saved
[19:22:15] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM1): All chunks are saved
[19:22:15] [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM-1): All chunks are saved
[19:22:15] [Server thread/INFO]: ThreadedAnvilChunkStorage: All dimensions are saved

For larger worlds with newly explored chunks, I've seen saving take up to 5 minutes!

Interestingly, Paper servers do not have this problem at all, their performance before and after 1.18.1 is fine. I imagine because they patch the chunk saving code.

Environment:

My server has 32GiB of RAM, 16 of which are free. 12 CPUs at about 20% usage. All this seems to suggest that the server has plenty of resources.

What now?

Certainly seems like a regression in 1.18.2+. If there is anything else I can do to help reproduce this, let me know.