[Topic-1] design: maxmemory ↔ compression interaction (R2.8.4) + R2.5.7 stickiness paragraph#36
Merged
Merged
Conversation
….7 stickiness paragraph R2.8.4 — savings are workload-dependent; existing observability is sufficient. Documents three interactions Topic-2 PR-B (the integration stress test) made it natural to revisit: - The same maxmemory setting yields different effective capacity depending on whether the workload is compressible (capacity grows by 1/ratio) or not (unchanged minus ~16 B/key header + fixed overhead). - Eviction interaction: per R2.8.1 the sampler sees compressed footprints, so eviction triggers at maxmemory measured in compressed bytes; victim selection by LRU/LFU is unchanged (no bias from per-key ratio asymmetry); evicted_keys is the canonical signal. - Decompress-mode drain causes used_memory to grow toward the no-compression baseline as the sweeper + permanent-decompress reclaim compressed frames; operators with active maxmemory should ensure headroom before flipping. Decision-record: no new compression-specific metric introduced for v1. PR-B's stress test surfaced no observation that would justify one. The existing fields plus the planned compression_transient_view_capped_total (S4.1) cover the operator-facing diagnostic surface. Revisit after S5.x benchmark scenarios. R2.5.7 — added stickiness paragraph cross-referenced from R2.5.6. Spells out that a value permanently decompressed (drain-mode or write-path mutation) stays RAW until either the sweeper visits it under master=compression (gated by compression-min-idle-seconds, by design to avoid worker-pool churn) or until eviction/expire/DEL makes the question moot. No immediate auto-recompress path; the sweeper pacing bounds the re-compress overhead globally. plan.md updated: - Topic-2 PR-B marked complete (merged at unstable@7013e370b via #35). - Topic-1 marked complete (lands via this same PR). Docs only — no code change. Build is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs-only PR — adds R2.8.4 (maxmemory ↔ compression interaction prose) + a stickiness paragraph in R2.5.7. No code change. Lands as Topic-1 from
implementation/plan.md, scheduled to follow Topic-2 PR-B so the prose could revisit whether anything new from PR-B's stress test justifies an additional metric.What changes
design/detailed-design.mdR2.8.4 — Savings are workload-dependent; existing observability is sufficient. New requirement under §2.8 Memory accounting. Three sub-points:
maxmemoryyields a wide range of effective dataset capacity:1/ratio.MEMORY USAGEsee compressed footprints. Eviction triggers atmaxmemorymeasured in compressed bytes; victim selection by LRU/LFU is unchanged (per-key compression-ratio asymmetry doesn't bias which key is picked);evicted_keys(existing) is the canonical signal.compression-master-switch decompressionprogressively reverts values to RAW (R2.5.7's permanent-decompress mode + sweeper).used_memorygrows toward the no-compression baseline. Operators with activemaxmemoryshould ensure headroom before flipping; otherwise the drain may trigger eviction.R2.5.7 — stickiness paragraph. Added inline (cross-referenced from R2.5.6). A value permanently decompressed — by drain-mode or by a write-path mutation through
compressionPermanentlyDecompress— stays RAW until one of:master=compressionand the eligibility predicate (R2.2) admits it. Thecompression-min-idle-secondsgate means a freshly-mutated value waits one idle window before re-compressing — by design, to avoid churning the worker pool on hot keys.No immediate auto-recompress path on permanent-decompress: re-compressing on every mutation would either churn the worker pool or compete for main-thread CPU; the sweep's pacing (
compression-sweep-max-cpu-pct) bounds the re-compress overhead globally.implementation/plan.md[x]). Notes the eventual merge atunstable@7013e370bvia fixup PR Ikolomi/dict import #35 — PR [Topic-2 PR-B] compression-stress.tcl integration tests + strEncoding fix #34 inadvertently merged into the now-staleikolomi/dict-importbase instead ofunstable, so the change set was re-landed via Ikolomi/dict import #35.[x]) — lands via this same PR.Decision-record: no new metric introduced for v1
A "compression headroom" metric (or some equivalent of "effective uncompressed capacity") was considered after Topic-2 PR-B's stress test landed and rejected. Reasoning:
used_memory(the standard one)compression_ratioandcompression_live_ratio_10mcompression_net_saved_bytescompression_compressed_objectscompression_total_uncompressed_bytes/compression_total_compressed_bytesevicted_keys(existing)compression_transient_view_capped_total(S4.1, R2.5.7) for cap-fallback observabilityRevisit after S5.x benchmark scenarios run if a specific operator-facing gap surfaces.
Verification
Docs-only — no code change.
git -P diff --statshows 2 files, +17 / -2 lines (15 new lines in detailed-design.md, 2 plan.md edits + 2 lines).CI's lint/clang-format/etc. should be unaffected.