Skip to content

[deps] Vendor ZSTD 1.5.5 for inline compression#5

Merged
GilboaAWS merged 1 commit into
unstablefrom
gilboa/vendor-zstd
May 18, 2026
Merged

[deps] Vendor ZSTD 1.5.5 for inline compression#5
GilboaAWS merged 1 commit into
unstablefrom
gilboa/vendor-zstd

Conversation

@GilboaAWS

@GilboaAWS GilboaAWS commented May 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Vendors the ZSTD 1.5.5 library source (lib/ directory only) under deps/zstd/ and wires it into the build system. This follows the same pattern as jemalloc, lua, and other vendored dependencies.

Review guidance

Only 2 files need review:

  • src/Makefile — the BUILD_ZSTD section (~6 lines)
  • deps/Makefile — new zstd target + distclean entry (~5 lines)

Everything under deps/zstd/ is unmodified upstream ZSTD 1.5.5 source (downloaded from github.com/facebook/zstd/releases/tag/v1.5.5, lib/ directory only). No need to review those ~100 files.

What changed

  • deps/zstd/: ZSTD 1.5.5 lib/ source — compression, decompression, and dictBuilder modules
  • deps/Makefile: new zstd target (builds libzstd.a), added to distclean
  • src/Makefile: BUILD_ZSTD=yes now adds zstd to DEPENDENCY_TARGETS, defines USE_ZSTD, includes deps/zstd headers, and links libzstd.a

Why

Phase 0 skeleton (PR #3) established the BUILD_ZSTD flag but deferred actual ZSTD linkage to Phase 1. The dictionary lifecycle (S1) and compression hot path (S2) implementations require ZSTD APIs (ZSTD_compress_usingCDict, ZSTD_decompress_usingDDict, ZDICT_trainFromBuffer, etc.). This PR unblocks that work.

Testing

  • make BUILD_ZSTD=yes — builds and links successfully
  • make BUILD_ZSTD=no — builds successfully (compression code compiles out)
  • ./runtest --single unit/type/compression — all 10 tests pass

@GilboaAWS GilboaAWS force-pushed the gilboa/vendor-zstd branch from 6c0b8ea to e56601a Compare May 18, 2026 08:22
@GilboaAWS GilboaAWS marked this pull request as draft May 18, 2026 08:31
@GilboaAWS GilboaAWS force-pushed the gilboa/vendor-zstd branch from e56601a to e1adedd Compare May 18, 2026 11:21
Add the ZSTD 1.5.5 lib/ directory (compression, decompression, and
dictBuilder modules) under deps/zstd/. Wire into the build system:

- deps/Makefile: new 'zstd' target builds libzstd.a
- src/Makefile: BUILD_ZSTD=yes adds zstd to DEPENDENCY_TARGETS,
  defines HAVE_ZSTD, includes deps/zstd headers, links libzstd.a
- BUILD_ZSTD=no continues to work (compression stubs compile out)

This unblocks Phase 1 implementation of the dictionary lifecycle (S1)
and compression hot path (S2) which require ZSTD APIs (ZSTD_compress,
ZSTD_decompress, ZDICT_trainFromBuffer, etc.).
@GilboaAWS GilboaAWS force-pushed the gilboa/vendor-zstd branch from e1adedd to 84cd986 Compare May 18, 2026 11:24
@GilboaAWS GilboaAWS marked this pull request as ready for review May 18, 2026 11:28
@GilboaAWS GilboaAWS merged commit 204420e into unstable May 18, 2026
80 checks passed
ikolomi added a commit that referenced this pull request May 24, 2026
Addresses 5 of 6 review comments on the QSBR design. Comment #6
(`compressionJob.key` extra-lookup concern) is explicitly deferred to
a follow-up PR per reviewer guidance.

Comment #1 (line 428) and #5 (line 544) — drop language-comparison
framing:
  Removed all references to Rust / `Arc<T>` / "memory-safe languages"
  / `shared_ptr` from §4.4 intro, the "Why QSBR" bullet list, and the
  §4.6 "Why the worker loads the active dict itself" paragraph. The
  rationale now stands on its own technical merit (decoupling the
  registry from worker hot paths; minimal worker contract;
  safe-directional failure modes) rather than via comparison to
  another language's type system. C with explicit protocols is the
  right tool for this problem; the comparison added rhetorical
  weight without adding signal.

Comment #2 (line 326) — duplication with R2.11.4:
  §3.3 Separation invariants restated the worker contract that
  R2.11.4 already specifies authoritatively. Slimmed the §3.3 bullet
  to a one-liner that points at §2.11 R2.11.4 and §4.4. Eliminates
  drift risk between the two places.

Comment #3 (line 439) — bound the retiring list, block on cap:
  Added new step 7 to the QSBR section explaining the cap interaction
  with R2.3.3. The retiring list is a subset of `dicts[]`, capped at
  `compression-dict-max-versions`. When grace-barrier draining cannot
  keep up (worker starvation, persistent `frame_refs > 0`), the cap
  is reached and BOTH training AND promotion are refused per R2.3.3:
  `LL_WARNING` log entry, `compression_dict_cap_reached` set in INFO,
  operator intervention required (raise cap or run COMPRESSION SWEEP).

Comment #4 (line 449) — grace-barrier wake-up via cond_broadcast:
  The original step 6 proposed enqueueing barrier jobs into the SPMC
  inbox to force idle workers to advance generations. This doesn't
  actually work: under work-stealing semantics a single fast worker
  can drain all barrier jobs while siblings stay asleep on the cond
  var. Rewrote step 6 to use a wake-all primitive built on
  `pthread_cond_broadcast`, and added a "Wake-all primitive"
  paragraph to §4.6 that describes extending `mutexqueue.h` with two
  new APIs: a broadcast wake-all (for QSBR grace barriers, config
  changes, etc.) and a shutdown-signal variant (for pool teardown).
  Step 6 cross-references §4.6 for the mechanism.

Comment #6 (line 513) — DEFERRED:
  Reviewer flagged that `compressionJob.key` (a `robj *` carried in
  the job) implies the main thread does an additional lookup at
  install time, doubling the per-write lookup cost. The reviewer
  explicitly tagged this as "follow up PR" — addressing it would
  require a redesign of the install-side data flow and is out of
  scope for the QSBR design change. Tracked as an open item; will
  be addressed before code lands for the install path (S2.7 in
  the implementation plan).
ikolomi added a commit that referenced this pull request Jun 10, 2026
…2.5.7

Implements the P1 semantics from the design doc revision in the
preceding commit. Five surgical changes plus a side bug fix:

1. Drop `applyCompressionEnabled` apply hook (config.c). The bool
   master switch is now a plain config flag with no side effects on
   toggle. The auto-sweep affordance is moved to an opt-in v2 config
   per Appendix D. Matches the `rdbcompression` config pattern.

2. Direction-conditional master-switch guard in `compressionSweep`
   command handler (compression.c). `direction=COMPRESS` requires
   the master switch on (otherwise rejected with a clear message
   pointing the operator at the only allowed direction).
   `direction=DECOMPRESS` is always allowed — it's the canonical
   drain path R2.1.4 calls out and operators MUST be able to invoke
   it after disabling.

3. Lift the unconditional master-switch early-return from
   `compressionSweepCron` (compression_sweep.c). Replace with
   state-machine-table semantics (§3.4):
     - IDLE.requested(COMPRESS) + master_off  → defensive clear (the
       command handler should already have rejected this; reaching
       cron means a future caller bypassed it)
     - SCANNING(COMPRESS) + master_off observed at cron entry  →
       abort, log NOTICE, return to IDLE. Honors R2.1.4's 'new writes
       stop being compressed' for background work too.
     - SCANNING(DECOMPRESS) + master_off  → continue scanning. The
       drain path is independent of the master switch.
   Worker-pool jobs already enqueued by sweepScanCallback before the
   abort still complete and install — workers don't observe the
   master switch (R2.4 / R2.11.4); the sweep state machine is the
   authoritative lever.

4. Add `compressionSweepCurrentDirection()` helper
   (compression_sweep.h + .c). Returns the in-flight sweep direction
   when SCANNING, 0 otherwise. Single int read, no atomics.
   Consumed by the two-mode drain in compressionBeforeSleep (#5).

5. Two-mode drain in `compressionBeforeSleep` (compression.c). Mode
   is selected once per invocation:
     - 'restore' (default): pointer-swap each side-map entry back
       to compressed (current behavior; R2.1.6 'master switch
       governs future behavior, existing state preserved').
     - 'permanent-decompress': fires when
       `compressionSweepIsScanning() && direction == DECOMPRESS`.
       Each side-map entry is finalized as RAW (compressed buffer
       released via the new releaseCompressedBuffer helper; temp
       sds stays installed). Cooperates with the operator-initiated
       drain.

6. Side improvement: extracted `releaseCompressedBuffer` helper
   that decodes the per-value header, decRef's the dict frame-ref,
   reverses the compression accounting, and frees the buffer. Used
   by:
     - compressionPermanentlyDecompress (refactored to use it)
     - compressionBeforeSleep permanent-decompress mode (new)
     - discardTransientEntry (BUG FIX — was previously `zfree`ing
       the buffer without decRef'ing the dict or reversing
       accounting; the dict-frame-ref leaked and
       compression_total_*_bytes drifted whenever the side-map
       discarded an orphaned entry)

Tests:
 - replaced `MasterSwitchOffMakesCronNoOp` (which asserted the
   pre-fix asymmetric behavior) with five P1-aligned tests:
     • NoAutoTriggerOnEnable
     • CompressRequestClearedAtCronWhenDisabled
     • DecompressSweepRunsWhenDisabled
     • InFlightCompressSweepAbortsOnDisable
     • InFlightDecompressSweepContinuesOnDisable
 - added BeforeSleepCooperatesWithDecompressSweep covering the
   two-mode drain end-to-end (compress, materialize transient view,
   tight-pace decompress sweep, beforeSleep, verify val_ptr is the
   temp sds and NOT swapped back to the compressed buffer)
 - added SweepCurrentDirectionReportsState exercising the new helper

Verified locally: server build clean (BUILD_ZSTD=yes default and
BUILD_ZSTD=no); Tcl unit/type/compression 11/11 pass; runtime smoke
of all five expected behaviors against a live server (toggle+sweep
matrix). Gtest unit tests not runnable locally (libgtest-dev not
installed); CI validates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants