[S2.5] Compression encoder path#17
Merged
Merged
Conversation
…rd in drain
Replaces the placeholder pass-through worker body with real
ZSTD_compress_usingCDict against the active dictionary, and replaces
the placeholder drain handler with the post-compression net-savings
guard (R2.4.3). The full install path (createCompressedObject +
dbOverwrite + compressionRegistryIncRef + caller-pin decRefCount)
remains stubbed — that lands with the write-path hook in S2.7 when
there's a real production caller emitting compression candidates.
Worker body (compression_workers.c):
- Per-worker ZSTD_CCtx allocated once at thread start, reused for
every job (CCtx is not thread-safe; one per worker keeps allocator
pressure off the per-job hot path).
- Atomically loads the active dict via compressionRegistryActive().
Per the QSBR contract (§4.4) the pointer stays valid until the
worker reports quiescent at end-of-iteration.
- "compression-enabled yes but no active dict yet" state (R2.1.5)
short-circuits to err=1 / dst=NULL — no allocation, no encode.
- Compresses directly into a single zmalloc'd buffer at offset
HEADER_SIZE; writes the header in-place after measuring the
actual compressed size; shrinks via zrealloc to avoid leaking
ZSTD_compressBound slack into used_memory (per design §5.2).
- Preserves ZSTD error codes via signed downcast; positive err
values are reserved for worker-policy decisions (no-dict, etc.).
Drain handler (compression_workers.c):
- err != 0 || dst == NULL: dispose, no install.
- dst_len >= uncompressed * (1 - savings_ratio_pct/100): net-savings
guard rejects, dispose. INFO counter
compression_skipped_incompressible++ comes in S4.1.
- Otherwise: install path is S2.7 (placeholder dispose for now).
Test-only accessors (compression_workers.h + .c):
- compressionWorkersDrainOutboxForTesting: peek without freeing.
- compressionWorkersFreeJobForTesting: caller-side cleanup.
- Field readers for compressionJob (file-private struct).
Production code consumes jobs via compressionWorkersDrainOutbox.
New gtest cases (test_compression_workers.cpp):
- RealCompressionRoundTrip: install synthetic dict, encode 1 KB
of compressible JSON, decompress with same DDict, byte-equal.
- NetSavingsGuardRejectsIncompressible: random bytes, drain through
production handler, verify no crash on rejection path.
- NoActiveDictMarksJobNotCompressed: enqueue without a dict, verify
err != 0 and dst == NULL.
- CompressionFromMultipleWorkersIsConsistent: 4 workers x 100 jobs,
each compressed buffer round-trips correctly.
Synthetic dict helper (installSyntheticDict, test-private): generates
~100 KB of training corpus from a small JSON-shaped sample list,
runs ZDICT_trainFromBuffer, creates CDict/DDict, adds via
compressionRegistryAdd. ~100x dict-size of training data is the ZSTD-
recommended minimum.
Verified locally:
- make -j2 -C src clean.
- ./runtest --single unit/type/compression: 10/10 pass.
- Server boots cleanly with --compression-enabled yes
--compression-threads 2 (worker pool starts, awaits jobs).
Not verified locally (CI):
- gtest unit tests (libgtest-dev not installed locally).
- clang-format-18 (not installed locally).
Plan.md S2.5 marked complete.
ikolomi
commented
Jun 3, 2026
ikolomi
commented
Jun 3, 2026
ikolomi
commented
Jun 3, 2026
ikolomi
commented
Jun 3, 2026
ikolomi
commented
Jun 3, 2026
Addresses the 5 review threads on PR #17. T-3346992511 (plan.md L174): drop "(PR pending)" parenthetical from the S2.5 sub-task. Eventually merged is the default state. T-3347191182 (drain handler): add TODO(S4.1) markers at the three drain-handler branches naming the exact counter contributions S4.1 will wire in. Per design R2.10.1 / R2.3.5: - err < 0 (real ZSTD error): compression_errors_total++ + rate- limited LL_WARNING per R6.1. - err > 0 (worker policy, no-dict): no counter — benign R2.1.5 state, tracked via compression_state. - Net-savings reject: compression_skipped_incompressible++ AND fold the actual measured ratio into compression_live_ratio_10m (rejection ratios in [0.9, 1.05] inflate the EMA and naturally trip the drift threshold per R2.3.5). - Successful install (S2.7+): compressions_per_sec rate update + fold success ratio into the EMA. Also added a TODO(S2.7) marker spelling out the install steps that land with the write-path hook. T-3347060548 (test seam in public header): refactor to match the established Valkey convention (see quicklist.c / intset.c testOnly* functions). - Renamed compressionWorkers*ForTesting → testOnlyCompressionWorkers*. - Removed the entire test-only block from compression_workers.h (~36 lines deleted). Production-callable surface now contains no test seams. - Definitions live only in compression_workers.c; the gtest test file declares what it needs locally in its own extern "C" block. - Replaced the 5 field accessors with a single testOnlyCompressionWorkersJobRead that fills caller-provided out-pointers. The result-projection struct (CompressionJobView) is defined in the gtest test file; production code carries no record of the projection layout. T-3347222201 (callout for @GilboaAWS): no code change; informational note about the synthetic-dict helper being portable to S1.x training tests. T-3347268445 (test:587 reject-path comment): prefixed the existing "comes in S4.1" comment with TODO(S4.1) so grep can find it when S4.1 wires the rejection counter. Convention adopted: TODO(<step-id>) comments mark code that intentionally defers behaviour to a future implementation step. Grep-friendly, ties deferred work to a specific plan step. Documented in SESSION_CHECKPOINT.md (working artifact). Verified locally: - make -j2 -C src clean. - ./runtest --single unit/type/compression: 10/10 pass. - No remaining ForTesting symbols anywhere in src/. Plan.md S2.5 line tightened.
…nclude
Addresses all CI failures on the previous push.
1. BUILD_ZSTD=no compatibility (build-32bit, test-ubuntu-latest-compression-off,
build-macos-latest, Analyze (cpp), test-sanitizer-address, etc. — same
root cause):
compression_workers.c made unconditional ZSTD calls that broke when
the parent is built without ZSTD. Gated all ZSTD usage behind
#ifdef USE_ZSTD:
- <zstd.h> include
- per-worker ZSTD_CCtx alloc/free at thread start/exit
- the encoder branch inside the per-job loop (the no-active-dict
branch is now also taken when USE_ZSTD is not compiled in)
When BUILD_ZSTD=no the worker pool runs but every job takes the
"not compressed" path, consistent with the registry being empty.
2. test_compression_workers.cpp BUILD_ZSTD=no compatibility:
- Wrapped <zstd.h> + <zdict.h> includes with #ifdef USE_ZSTD.
- Wrapped the entire S2.5 encoder-path test section + helpers
(installSyntheticDict, decompressBuffer) with #ifdef USE_ZSTD.
- Added #include "compression_header.h" so COMPRESSION_HEADER_SIZE
is defined (root cause of the test-ubuntu-latest compile error).
3. unit/Makefile: propagate USE_ZSTD to the gtest build:
- Detect -DUSE_ZSTD in PREV_FINAL_CFLAGS (set by parent's
BUILD_ZSTD=yes via .make-settings).
- Adds ZSTD_CFLAGS = -DUSE_ZSTD=1 -I../../deps/zstd to the C++
compile rule, mirroring the existing JEMALLOC_CFLAGS pattern.
- Drops ZSTD_LIB from LD_LIBS when USE_ZSTD is not propagated
(libzstd.a is not built when BUILD_ZSTD=no, so the link
referencing it would fail).
4. clang-format-check: three trivial fixes flagged by clang-format-18:
- Trailing-comment alignment in compression_workers.c (off by one
space after a column shift).
- Extra space in `int testOnly...` declaration in test file.
- Single-line `if (...) { ... }` in test file expanded to
multi-line per the project style.
Verified locally:
- make -j2 -C src clean (BUILD_ZSTD=yes default).
- make -j2 -C src BUILD_ZSTD=no clean.
- ./runtest --single unit/type/compression: 10/10 pass under both
BUILD_ZSTD=yes and BUILD_ZSTD=no.
CI cells expected to recover (all share the same root causes above):
- clang-format-check
- build-32bit
- build-macos-latest
- test-ubuntu-latest
- test-ubuntu-latest-cmake-tls
- test-ubuntu-latest-compression-off
- test-ubuntu-latest-compatibility (7.2.11, 8.0.6, 8.1.4)
- test-sanitizer-address
- Analyze (cpp) [CodeQL]
The 4 still-failing CI cells (test-ubuntu-latest, test-sanitizer-address, build-32bit, code-coverage) all hit the same 3 gtest failures: FAILED: CompressionWorkersTest.RealCompressionRoundTrip FAILED: CompressionWorkersTest.NetSavingsGuardRejectsIncompressible FAILED: CompressionWorkersTest.CompressionFromMultipleWorkersIsConsistent Server log on the failing tests: Compression: dictionary registry cap reached (0). Run COMPRESSION SWEEP or raise compression-dict-max-versions. Root cause: server.compression_dict_max_versions defaults to 0 in the gtest harness (no real config layer), and the registry rejects every promotion attempt when the cap is 0. installSyntheticDict() therefore returns dict_id=0, and the tests' ASSERT_NE(dict_id, 0u) fails. Mirrors what test_compression_registry.cpp already does in its SetUp (line 29 sets the same field to 4). Added matching initialization to the CompressionWorkersTest::SetUp. Verified locally: - make -j2 -C src clean. - ./runtest --single unit/type/compression: 10/10 pass. build-debian-old failure is unrelated and genuinely flaky (apt fetched a debian package and got "Connection reset by peer") — will self-resolve on retry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the S2.4 placeholder pass-through worker body with real
ZSTD_compress_usingCDict, and replaces the placeholder drain handler with the post-compression net-savings guard (R2.4.3). The full install path (createCompressedObject+dbOverwrite+compressionRegistryIncRef+ caller-pindecrRefCount) remains stubbed — that lands with the write-path hook in S2.7 when there's a real production caller emitting compression candidates.Commits in this PR
c298fbdb3b5c9773a333a1e1f3aBUILD_ZSTD=nocompatibility (gated all ZSTD usage with#ifdef USE_ZSTD),unit/Makefilepropagates-DUSE_ZSTDand-I../deps/zstdto the gtest build, clang-format-18 fixes, missing#include "compression_header.h"in test0fd814c9eserver.compression_dict_max_versionsin the test fixture (mirrors whattest_compression_registry.cppalready does); fixes the runtime gtest failuresArchitecture
ZSTD_CCtxper worker thread, allocated at start, reused for every jobZSTD_CCtxis not thread-safe (zstd API contract); per-thread allocation keeps allocator pressure off the per-job hot pathcompressionRegistryActive()per jobzmallocsdst, drain handlerzfreesrobj(R2.11.4)[16-byte header][ZSTD frame]buf + HEADER_SIZE, write header in-place after measuring; matches the zero-copy contract documented incompression_header.hzreallocto actual size after compressionZSTD_compressBoundover-allocation intoused_memory(per design §5.2)job->erris signed; ZSTD errors → negative values, worker-policy → positive values (err=1= no active dict)BUILD_ZSTD=nomode#ifdef USE_ZSTD; worker takes the no-active-dict branch unconditionallytestOnly*symbols defined in.conly, not declared in the public headerquicklist.c/intset.c) — production-callable surface stays free of test entry pointsWhat this PR does NOT do
dbAdd/dbSetValue/dbOverwriteobjectGetUncompressedViewfor read pathscreateCompressedObject+dbOverwrite+ dict frame-ref accounting + caller-pindecrRefCountINFO compressioncounters:compression_skipped_incompressible,compression_errors_total,compression_live_ratio_10m,compression_compressions_per_secTODO(S4.1):markers at the three branches naming each contributionWorker body — key decisions
The
if (shrunk == NULL)defensive path keeps the originalbufand proceeds;zreallocon shrink is allocator-specific behavior. We pay the slack cost rather than fail the job.Drain handler — three branches
The integer-form net-savings comparison avoids floating point.
job->dst_lenalready includes the 16-byte header, so it IS the on-heap footprint we want to compare against.Test seam (refactored after PR review)
Production code carries no test-only symbols in its public surface. Following the established Valkey convention (see
quicklist.c/intset.c):The compressionJob struct stays file-private. The single
testOnlyCompressionWorkersJobReadfunction projects its fields into caller-provided out-pointers; the gtest test code defines its own flatCompressionJobViewstruct that the read function fills.Tests
src/unit/test_compression_workers.cpp— 4 new gtest cases (gated behind#ifdef USE_ZSTD):RealCompressionRoundTripNetSavingsGuardRejectsIncompressibleTODO(S4.1):extends to assert counter increments.NoActiveDictMarksJobNotCompressederr != 0anddst == NULL(R2.1.5 path)CompressionFromMultipleWorkersIsConsistentThe synthetic-dict helper (
installSyntheticDict) generates ~100 KB of training corpus (1500 samples × ~70 B) — ZSTD's documented minimum is ~100x dict size. Goes through the production registry add/promote path; only the corpus source differs from real training (S1.2 plumbs main-thread kvstore iteration + bio).The 14 existing gtest cases from S2.4 stayed encoder-agnostic — they assert only on job count and thread-count transitions, so they continue passing without modification (verified by running
./runtest --single unit/type/compression).Build-system change
src/unit/Makefilenow propagates-DUSE_ZSTD=1 -I../../deps/zstdto the gtest build (mirroring the existingJEMALLOC_CFLAGSpattern). Conditional on-DUSE_ZSTDbeing present inPREV_FINAL_CFLAGS(which the parent'sBUILD_ZSTD=yessets via.make-settings). Without ZSTD,ZSTD_LIBis dropped fromLD_LIBSso the gtest binary doesn't try to link a non-existentlibzstd.a.Verified locally
make -j2 -C srcclean (BUILD_ZSTD=yes default).make -j2 -C src BUILD_ZSTD=noclean../runtest --single unit/type/compression— 10/10 transparency tests pass under bothBUILD_ZSTD=yesandBUILD_ZSTD=no.--compression-enabled yes --compression-threads 2.Not verified locally (CI gates)
libgtest-devnot installed in this dev environment).clang-format-18.Diff stat