Streaming save and memory reduction #699
Conversation
|
⏱️ Benchmark run finished in 0m 27s 📈 Compared against baseline: 30s 🟢 Generation time is unchanged. 📅 Last benchmark: 2026-01-06 22:20:24 UTC You can retrigger the benchmark by commenting |
There was a problem hiding this comment.
Pull request overview
This PR implements streaming save mode and memory reduction optimizations for large Minecraft world generation. The main changes focus on reducing peak memory usage when processing and saving large worlds (100+ regions) by processing data sequentially and releasing memory as soon as elements/regions are processed.
Key Changes:
- Converted parallel region saving to sequential streaming mode that releases memory after each region
- Introduced deterministic RNG to ensure consistent random generation across region boundaries
- Implemented memory-efficient element processing that drops elements immediately after use
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/world_editor/java.rs | Refactored from parallel (par_iter) to sequential region saving with memory release after each region; extracted save_single_region helper method |
| src/deterministic_rng.rs | New module providing seeded RNG functions (element_rng, coord_rng) for consistent randomness across region boundaries; includes comprehensive unit tests |
| src/data_processing.rs | Changed element processing from iteration to consumption via pop() for immediate memory release; added flood fill cache cleanup after each element |
| src/floodfill_cache.rs | Added remove_way and remove_relation_ways methods for progressive cache cleanup; removed thread pool configuration log output |
| src/element_processing/*.rs | Updated tree, natural, leisure, landuse, buildings, and amenities modules to use deterministic RNG instead of thread_rng() |
| src/element_processing/water_areas.rs | Removed unused start_time parameter and timeout checking code (now commented out) |
| src/main.rs | Added deterministic_rng module import |
| Cargo.toml / Cargo.lock | Added rand_chacha dependency for deterministic RNG |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
retrigger-benchmark |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
No description provided.