Halve LCC2 read peak memory by decoding chunks into a preallocated table#260
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces peak memory usage when reading LCC2 scenes by switching from “decode all chunks then combine()” to a deterministic preallocate-and-copy pipeline, while also improving related ergonomics (progress logging, tolerant meta parsing) and eliminating repeated WebP WASM instantiation overhead.
Changes:
- Precompute per-chunk counts (from meta when available; otherwise via cheap header/meta reads) and decode chunks directly into preallocated output arrays with deterministic offsets.
- Memoize the WebP WASM module in
WebPCodec.create()and update SOG/image writers to rely on the memoization. - Add progress-bar control for SOG decoding, safer trailing-comma JSON parsing, chunk index validation, and expanded unit tests for LCC2/WebP paths.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/webp-codec.test.mjs | Adds coverage for WebP WASM module memoization (sequential + concurrent) and a lossless round-trip. |
| test/read-lcc2.test.mjs | Adds extensive LCC2 tests for meta parsing, chunk count fallbacks, deterministic assembly, heterogeneous columns, and index validation. |
| src/lib/writers/write-sog.ts | Removes per-module codec instance cache; relies on WebPCodec.create() module memoization. |
| src/lib/writers/write-image.ts | Removes per-module codec instance cache; relies on WebPCodec.create() module memoization. |
| src/lib/utils/webp-codec.ts | Memoizes the compiled/instantiated WASM module promise and resets cache on rejection. |
| src/lib/readers/read-sog.ts | Adds logging option to suppress internal progress bars for callers doing concurrent chunk reads. |
| src/lib/readers/read-sog-v1.ts | Threads through ReadSogOptions to support silent progress mode for v1 as well. |
| src/lib/readers/read-lcc2.ts | Implements preallocated output decoding, count scanning/header parsing, safer JSON parsing, single traversal chunk collection, and index validation. |
Comments suppressed due to low confidence (1)
src/lib/readers/read-lcc2.ts:798
envFileIndexis derived from meta.root.data.env.name but isn’t validated before indexing intosplatFiles. If it’s out of range or points to an empty path,related(splatFiles[envFileIndex])can throw (or later decode will fail) and the error is currently swallowed as a warn in the optional-env path. This undermines the new “chunk file index validation” behavior; invalid env references should fail fast with a descriptive error (or be handled explicitly).
if (envFileIndex !== undefined) {
try {
const envFull = related(splatFiles[envFileIndex]);
const envTable = await decodeChunk(fileSystem, splatType, envFull);
envTable.addColumn(
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reading an LCC2 scene currently holds every decoded chunk DataTable in memory and then
combine()allocates a second full-size copy, so peak memory is ~2× the decoded scene (~53GB for a real 110M-splat scene with 3-band SH). This PR replaces decode-then-combine with a preallocate-and-copy scheme and fixes several smaller inefficiencies found in the same path.Changes
data['3dgs'].countfields (summed across nodes sharing a chunk file, matching real XGrids exports). When a count is missing (old-protocol files), a cheap fallback reads just the chunk header: SOGmeta.jsoncount/means.shape[0], or SPZnumPoints(including streaming-gunzip of just the 16-byte header for gzip-wrapped v1–3 containers). Prefix-summed offsets fix the output layout before decoding, so workers completing out of order still write into disjoint regions and the result stays deterministic. Each decoded chunk is validated against its expected count, copied into the output arrays at its offset and released. Output columns are allocated once at exact final size on first sighting, preservingcombine()'s zero-fill semantics for heterogeneous chunks. Peak memory is now ~1× the scene plus the chunks in flight.WebPCodec.create()previously compiled and instantiated a fresh Emscripten module (with its own heap) on every call — once per SOG chunk viareadSog. The module promise is now cached, mirroring the existinggetSpzModule()pattern.readSogcalls each opening its own bar, corrupting the display (sibling bars popped as failed, outer bar suppressed).readSog/readSogV1now accept{ logging: 'own' | 'silent' }(mirroringwriteSog) and readLcc2 silences the per-chunk bars under its own single bar.collectChunksByLevelgathers every level's chunk files (with counts) in one pass, replacing the per-LODcollectFileIndicesForLodtraversals.JSON.parseis tried first; only on failure is a string-aware trailing-comma stripper applied. The old,\s*}regex corrupted string values containing",}"and missed trailing commas before].splatFilesreferences now fail with a descriptive error instead of crashing injoin().Verification
nulloutput): LOD 3 (6.8M splats) 3.50GB → 2.51GB, LOD 2 (13.7M splats) 6.71GB → 4.48GB, with run time at parity. The gap grows with scene size since the overhead beyond 1× is now a constant ~LOAD_CONCURRENCYchunks.