[deps] Vendor ZSTD 1.5.5 for inline compression#5
Merged
Conversation
6c0b8ea to
e56601a
Compare
e56601a to
e1adedd
Compare
Add the ZSTD 1.5.5 lib/ directory (compression, decompression, and dictBuilder modules) under deps/zstd/. Wire into the build system: - deps/Makefile: new 'zstd' target builds libzstd.a - src/Makefile: BUILD_ZSTD=yes adds zstd to DEPENDENCY_TARGETS, defines HAVE_ZSTD, includes deps/zstd headers, links libzstd.a - BUILD_ZSTD=no continues to work (compression stubs compile out) This unblocks Phase 1 implementation of the dictionary lifecycle (S1) and compression hot path (S2) which require ZSTD APIs (ZSTD_compress, ZSTD_decompress, ZDICT_trainFromBuffer, etc.).
e1adedd to
84cd986
Compare
ikolomi
approved these changes
May 18, 2026
ikolomi
added a commit
that referenced
this pull request
May 24, 2026
Addresses 5 of 6 review comments on the QSBR design. Comment #6 (`compressionJob.key` extra-lookup concern) is explicitly deferred to a follow-up PR per reviewer guidance. Comment #1 (line 428) and #5 (line 544) — drop language-comparison framing: Removed all references to Rust / `Arc<T>` / "memory-safe languages" / `shared_ptr` from §4.4 intro, the "Why QSBR" bullet list, and the §4.6 "Why the worker loads the active dict itself" paragraph. The rationale now stands on its own technical merit (decoupling the registry from worker hot paths; minimal worker contract; safe-directional failure modes) rather than via comparison to another language's type system. C with explicit protocols is the right tool for this problem; the comparison added rhetorical weight without adding signal. Comment #2 (line 326) — duplication with R2.11.4: §3.3 Separation invariants restated the worker contract that R2.11.4 already specifies authoritatively. Slimmed the §3.3 bullet to a one-liner that points at §2.11 R2.11.4 and §4.4. Eliminates drift risk between the two places. Comment #3 (line 439) — bound the retiring list, block on cap: Added new step 7 to the QSBR section explaining the cap interaction with R2.3.3. The retiring list is a subset of `dicts[]`, capped at `compression-dict-max-versions`. When grace-barrier draining cannot keep up (worker starvation, persistent `frame_refs > 0`), the cap is reached and BOTH training AND promotion are refused per R2.3.3: `LL_WARNING` log entry, `compression_dict_cap_reached` set in INFO, operator intervention required (raise cap or run COMPRESSION SWEEP). Comment #4 (line 449) — grace-barrier wake-up via cond_broadcast: The original step 6 proposed enqueueing barrier jobs into the SPMC inbox to force idle workers to advance generations. This doesn't actually work: under work-stealing semantics a single fast worker can drain all barrier jobs while siblings stay asleep on the cond var. Rewrote step 6 to use a wake-all primitive built on `pthread_cond_broadcast`, and added a "Wake-all primitive" paragraph to §4.6 that describes extending `mutexqueue.h` with two new APIs: a broadcast wake-all (for QSBR grace barriers, config changes, etc.) and a shutdown-signal variant (for pool teardown). Step 6 cross-references §4.6 for the mechanism. Comment #6 (line 513) — DEFERRED: Reviewer flagged that `compressionJob.key` (a `robj *` carried in the job) implies the main thread does an additional lookup at install time, doubling the per-write lookup cost. The reviewer explicitly tagged this as "follow up PR" — addressing it would require a redesign of the install-side data flow and is out of scope for the QSBR design change. Tracked as an open item; will be addressed before code lands for the install path (S2.7 in the implementation plan).
ikolomi
added a commit
that referenced
this pull request
Jun 10, 2026
…2.5.7
Implements the P1 semantics from the design doc revision in the
preceding commit. Five surgical changes plus a side bug fix:
1. Drop `applyCompressionEnabled` apply hook (config.c). The bool
master switch is now a plain config flag with no side effects on
toggle. The auto-sweep affordance is moved to an opt-in v2 config
per Appendix D. Matches the `rdbcompression` config pattern.
2. Direction-conditional master-switch guard in `compressionSweep`
command handler (compression.c). `direction=COMPRESS` requires
the master switch on (otherwise rejected with a clear message
pointing the operator at the only allowed direction).
`direction=DECOMPRESS` is always allowed — it's the canonical
drain path R2.1.4 calls out and operators MUST be able to invoke
it after disabling.
3. Lift the unconditional master-switch early-return from
`compressionSweepCron` (compression_sweep.c). Replace with
state-machine-table semantics (§3.4):
- IDLE.requested(COMPRESS) + master_off → defensive clear (the
command handler should already have rejected this; reaching
cron means a future caller bypassed it)
- SCANNING(COMPRESS) + master_off observed at cron entry →
abort, log NOTICE, return to IDLE. Honors R2.1.4's 'new writes
stop being compressed' for background work too.
- SCANNING(DECOMPRESS) + master_off → continue scanning. The
drain path is independent of the master switch.
Worker-pool jobs already enqueued by sweepScanCallback before the
abort still complete and install — workers don't observe the
master switch (R2.4 / R2.11.4); the sweep state machine is the
authoritative lever.
4. Add `compressionSweepCurrentDirection()` helper
(compression_sweep.h + .c). Returns the in-flight sweep direction
when SCANNING, 0 otherwise. Single int read, no atomics.
Consumed by the two-mode drain in compressionBeforeSleep (#5).
5. Two-mode drain in `compressionBeforeSleep` (compression.c). Mode
is selected once per invocation:
- 'restore' (default): pointer-swap each side-map entry back
to compressed (current behavior; R2.1.6 'master switch
governs future behavior, existing state preserved').
- 'permanent-decompress': fires when
`compressionSweepIsScanning() && direction == DECOMPRESS`.
Each side-map entry is finalized as RAW (compressed buffer
released via the new releaseCompressedBuffer helper; temp
sds stays installed). Cooperates with the operator-initiated
drain.
6. Side improvement: extracted `releaseCompressedBuffer` helper
that decodes the per-value header, decRef's the dict frame-ref,
reverses the compression accounting, and frees the buffer. Used
by:
- compressionPermanentlyDecompress (refactored to use it)
- compressionBeforeSleep permanent-decompress mode (new)
- discardTransientEntry (BUG FIX — was previously `zfree`ing
the buffer without decRef'ing the dict or reversing
accounting; the dict-frame-ref leaked and
compression_total_*_bytes drifted whenever the side-map
discarded an orphaned entry)
Tests:
- replaced `MasterSwitchOffMakesCronNoOp` (which asserted the
pre-fix asymmetric behavior) with five P1-aligned tests:
• NoAutoTriggerOnEnable
• CompressRequestClearedAtCronWhenDisabled
• DecompressSweepRunsWhenDisabled
• InFlightCompressSweepAbortsOnDisable
• InFlightDecompressSweepContinuesOnDisable
- added BeforeSleepCooperatesWithDecompressSweep covering the
two-mode drain end-to-end (compress, materialize transient view,
tight-pace decompress sweep, beforeSleep, verify val_ptr is the
temp sds and NOT swapped back to the compressed buffer)
- added SweepCurrentDirectionReportsState exercising the new helper
Verified locally: server build clean (BUILD_ZSTD=yes default and
BUILD_ZSTD=no); Tcl unit/type/compression 11/11 pass; runtime smoke
of all five expected behaviors against a live server (toggle+sweep
matrix). Gtest unit tests not runnable locally (libgtest-dev not
installed); CI validates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vendors the ZSTD 1.5.5 library source (lib/ directory only) under
deps/zstd/and wires it into the build system. This follows the same pattern as jemalloc, lua, and other vendored dependencies.Review guidance
Only 2 files need review:
src/Makefile— theBUILD_ZSTDsection (~6 lines)deps/Makefile— newzstdtarget +distcleanentry (~5 lines)Everything under
deps/zstd/is unmodified upstream ZSTD 1.5.5 source (downloaded from github.com/facebook/zstd/releases/tag/v1.5.5, lib/ directory only). No need to review those ~100 files.What changed
deps/zstd/: ZSTD 1.5.5 lib/ source — compression, decompression, and dictBuilder modulesdeps/Makefile: newzstdtarget (buildslibzstd.a), added todistcleansrc/Makefile:BUILD_ZSTD=yesnow addszstdtoDEPENDENCY_TARGETS, definesUSE_ZSTD, includesdeps/zstdheaders, and linkslibzstd.aWhy
Phase 0 skeleton (PR #3) established the
BUILD_ZSTDflag but deferred actual ZSTD linkage to Phase 1. The dictionary lifecycle (S1) and compression hot path (S2) implementations require ZSTD APIs (ZSTD_compress_usingCDict,ZSTD_decompress_usingDDict,ZDICT_trainFromBuffer, etc.). This PR unblocks that work.Testing
make BUILD_ZSTD=yes— builds and links successfullymake BUILD_ZSTD=no— builds successfully (compression code compiles out)./runtest --single unit/type/compression— all 10 tests pass