# Conflicts:
#	src/io_threads.c
#	src/networking.c
#	src/replication.c

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

The compression CI job (--config replcompression yes) ran replication-buffer.tcl
and the dual-channel buffer-memory tests, which assert exact replication
buffer/backlog memory and byte volumes. Compression legitimately changes those
(per-replica codec buffers add ~1MB scratch; fewer wire bytes let replicas keep
up), so the assertions fail under compression even though replication is
correct. Drop the repl-compression tag from replication-buffer.tcl and the two
dual-channel blocks holding the memory tests; they still run uncompressed in the
regular job. Functional dual-channel coverage stays in the compression job.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…io#3897)

## Summary

Fixes valkey-io#3008

> lets add assert checking that the object has a key in
dbUntrackKeyWithVolatileItems and dbTrackKeyWithVolatileItems to be able
to get a more explicit error in these cases

This is addressed in this PR.

Signed-off-by: ydsakshi <ydsakshi023@gmail.com>

clusterNode.shard_id is a fixed-size char[CLUSTER_NAMELEN] buffer
that is not guaranteed to be NUL-terminated, so it must be printed
with %.40s.

This was introduced in valkey-io#2510.

Signed-off-by: Binbin <binloveplay1314@qq.com>

…valkey-io#3941)

Since valkey-io#2449 made the failover delay relative to cluster-node-timeout.
Now delay = min(cluster-node-timeout / 30, 500), any cluster-node-timeout
below 30, including the legal minimum 0 will collapses delay to zero,
and `x % 0` is undefined behaviour.

Signed-off-by: Binbin <binloveplay1314@qq.com>

…thakaggarwal97/valkey into replication-streaming-compression-pr

# Conflicts:
#	src/compression_stream.c
#	src/compression_stream.h
#	src/config.c
#	src/rdb.c
#	src/server.h
#	src/unit/test_compression.cpp
#	tests/integration/rdb-compression.tcl
#	valkey.conf

…tate and plaintext passthrough

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…alkey-io#3938)

This regression is still present in 9.1 GA as the cherrypick of revert
commit was missed during release.

Re-applies valkey-io#3544 (reverted in valkey-io#3756 due to ~20% SET regression) with the
performance fix from valkey-io#3760.

**Root Cause:** The original valkey-io#3544 changed
`tryOffloadFreeObjToIOThreads` to only offload the SDS buffer free to IO
threads, freeing the `robj` shell on the main thread. I carried out
profiling for the change and it showed that freeing the `robj` shell on
the main thread became the prime main-thread hotspot (~10% CPU), while
IO threads shifted from doing real `jemalloc` work to spinning idle on
`spmcDequeue`.

**Fix**: Keep `tryOffloadFreeObjToIOThreads` offloading the entire robj
(`decrRefCount`) to the IO thread. Cross-thread `zfree` is safe with
`jemalloc`.

This PR includes all cleanup work from valkey-io#3544 so - 
- `trySendWriteToIOThreads`: defer clearing `last_header` until after
successful enqueue
- `evictClients`: simplified bookkeeping
- Queue sizes as runtime parameters instead of compile-time macros
- IO ignition policy using `stat_active_time` instead of `getrusage`
- Function renames (`IOThreadFreeArgv` --> `ioThreadFreeArgv`, etc.) and
doc comments

**Benchmark** on (Graviton4 c8gb.metal-48xl):
Config: SET, 128B values, 9 IO threads, pipeline=10, 1600 clients - Same
as Valkey official method
| Version | Throughput |
|---------|-----------|
| Unstable + original valkey-io#3544 | ~1,554K rps |
| Unstable + this PR | ~2,116K rps |

<details>
<summary>Diff vs original valkey-io#3544 (perf fix)</summary>

```diff
diff --git a/src/io_threads.c b/src/io_threads.c
--- a/src/io_threads.c
+++ b/src/io_threads.c
@@ // IO thread handler
 case JOB_REQ_FREE_OBJ:
-    zfree(data);
+    decrRefCount(data);
     break;

@@ // tryOffloadFreeObjToIOThreads
-    /* We offload only the free of the ptr that may be allocated by the I/O thread.
-     * The object itself was allocated by the main thread and will be freed by the main thread. */
-    void *job = tagJob(sdsAllocPtr(objectGetVal(obj)), JOB_REQ_FREE_OBJ);
+    void *job = tagJob(obj, JOB_REQ_FREE_OBJ);
     if (unlikely(spmcEnqueue(&io_shared_inbox, job) == false)) return C_ERR;
-    objectSetVal(obj, NULL);
-    decrRefCount(obj);
     io_jobs_submitted++;
```
</details>

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…val (fix macOS -Werror)

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…3949)

This test runs `start_cluster 4 5` and manually attaches an extra
replica with `CLUSTER REPLICATE`, so one primary ends up with two
replicas while the others have one. It then pauses and restarts all 8
nodes and asserts that every node's shard id is unchanged.

This is why the test fails:
- A primary is termed "orphaned" when it owns slots and is flagged
`MIGRATE_TO`, but currently has zero working replicas.
- When a primary is orphaned, the cluster moves a spare replica to it. A
replica is only spare if its current primary would still have more than
`cluster-migration-barrier` replicas after it leaves (default barrier is
1).
- A migrating replica adopts the target primary's shard id, so its shard
id changes.

During the staggered restart, a primary can be left with no reachable
replica for a short window and become orphaned. Once the cluster returns
to OK, the primary that has two replicas is the only one eligible to
donate, so one of its replicas migrates to the orphaned primary and
takes on its shard id.

The fix is to set `cluster-allow-replica-migration no` on this block so
replicas stay with their original primaries across the restart. Replica
migration is not what this test is checking.

Fixes valkey-io#3914

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

…io#3947)

The latency report told users to run 'echo never > .../enabled' to
disable THP, but checkTHPEnabled recommends 'echo madvise' and notes
that 'madvise' or 'never' are both valid. Make createLatencyReport
consistent: suggest 'madvise' in the command and mention both values
in the accompanying note.

This advice was originally added in 1461f02.

Signed-off-by: Binbin <binloveplay1314@qq.com>

…et (valkey-io#3950)

When a hash has many fields whose expirations fall in the same
volatile-set time-bucket (>127 entries), that bucket is encoded as a
hashtable (HT). The active field-expiry path drains expired entries via
vsetBucketRemoveExpired_HASHTABLE(), but it is bounded by the per-key
expire quota (max_count, from dbReclaimExpiredFields). When the quota
stops the drain one entry short, the function only collapsed the bucket
to NONE at size 0 and left it as an HT bucket holding a single entry.

That violates the invariant enforced by removeFromBucket_HASHTABLE(),
which asserts hashtableSize(ht) > 0 after a delete and downgrades an HT
bucket to SINGLE at size 1. A later normal removal of that lone field --
HDEL, or a cross-bucket HSETEX update routed through
removeEntryFromRaxBucket() -- deletes the sole entry, drops the size 1
-> 0, and trips the assertion.

Fix: Apply the same downgrade in the expiry path: when draining leaves
exactly one entry, downgrade the HT bucket to SINGLE so the invariant
holds. Behavior is unchanged when the bucket drains fully (-> NONE) or
retains two or more entries (-> HT, as the normal removal path already
tolerates).

Add a regression test that fills one time-bucket past the HT threshold,
expires all but one entry, then removes the survivor through the normal
path; it crashed before this fix and passes after.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

…er of each shard (valkey-io#3946)

We previously attempted to set `cluster-node-timeout` to 15000 in valkey-io#2793
but failed. This was because we did not explicitly specify it and relied
on the default value, but `start_cluster` internally sets it to 3000.

Closes valkey-io#3932.

Signed-off-by: Binbin <binloveplay1314@qq.com>

valkey-io#3964)

Database-level ACL valkey-io#2309 introduced `alldbs` rule that was explicit for
all users and because of that previous versions no longer had the
ability to parse ACL strings produced by later versions.
Omit `alldbs` in `ACLDescribeSelector()`, that is used in `ACL
SAVE/LOAD` and `CONFIG REWRITE` command paths so that downgrades would
be possible if new feature was not used (`db=` and `resetdbs` rules).
Keep `ACL GETUSER` command's output as is and return `alldbs` in
`databases` field because of command's field-value format.
Add test to check that `ACL LIST` omits implicit `alldbs` and add check
to existing `ACL SAVE` and `CONFIG REWRITE` tests.

Fixes valkey-io#3915

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>

Sarthak Aggarwal (@sarthakaggarwal97) has been granted write permissions
by maintainer consensus. This adds him to the Current Committers list in
MAINTAINERS.md.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…alkey-io#3920)

## Problem

A crafted zipmap entry can set the value length to a value near
`UINT32_MAX` so that the `l + e` sum (value length + one-byte free
space) wraps in `unsigned int` arithmetic. The wrapped sum advances the
validation cursor by a tiny amount, leaving `p` inside the buffer, so
the `OUT_OF_RANGE` check passes and `zipmapValidateIntegrity` wrongly
returns success. The field-length path has the same shape — advancing
`p` by a ~4GB length wraps the pointer on 32-bit builds.

`zipmapValidateIntegrity` is always called with `deep=1` from `rdb.c`
when loading `RDB_TYPE_HASH_ZIPMAP`, **including via `RESTORE`**, so any
client with `RESTORE` access can submit a payload that passes
validation. On 32-bit platforms this leads to out-of-bounds access
during the subsequent zipmap→listpack conversion. On 64-bit the
downstream `lpSafeToAdd` cap happens to reject it (the raw ~4GB length
exceeds `LISTPACK_MAX_SAFETY_SIZE`), but the validator should not accept
a malformed payload in the first place — this is the function whose sole
job is to reject it.

## Fix

Bounds-check the attacker-controlled length against the bytes remaining
in the zipmap, in 64-bit space, **before** any pointer arithmetic, for
both the field-length and value-length paths.

## Testing

- `tests/integration/corrupt-dump.tcl`: a `RESTORE`-path test exercising
the full attack surface; asserts rejection and that the server stays up.
- Verified the test **fails on the pre-fix code** (validator accepts the
value-length payload) and **passes after the fix**, confirmed by
stashing the fix during the integration run.
- Full `integration/corrupt-dump` suite: 76 passed, 0 failed.

> [!NOTE]
> Found via structure-aware fuzzing of the RESTORE path. This issue was
generated by AI but verified, with love, by a human.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

…valkey-io#3921)

## Problem

A crafted `RESTORE` payload can store a `NAN` score in a
listpack-encoded sorted set. The integrity validation
(`lpValidateIntegrityAndDups`) only checks the listpack *structure* and
member uniqueness — it does not check score validity — so the payload is
accepted on load.

When the sorted set is later converted to a skiplist (e.g. when it grows
past `zset-max-listpack-entries`, or via any operation that triggers
conversion), `zslInsertNode()` asserts the score is not `NAN`
(`t_zset.c:260`) and the server aborts. **Any client with `RESTORE`
access can remotely crash the server.**

The skiplist RDB format (`RDB_TYPE_ZSET` / `RDB_TYPE_ZSET_2`) already
rejects `NAN` scores at load time (`rdb.c`, "Zset with NAN score
detected"). The listpack format (`RDB_TYPE_ZSET_LISTPACK`) had no
equivalent check.

## Reproduction

```
RESTORE k 0 "\x11\x19\x19\x00\x00\x00\x04\x00\x82m1\x03\x83nan\x04\x82m2\x03\x832.5\x04\xFF\x50\x00...."
# loads OK, then:
ZADD k 9 x      # forces listpack->skiplist conversion -> serverAssert(!isnan(node->score)) -> SIGABRT
```

## Fix

Add `zzlValidateScores()`, which scans the scores of a listpack zset
after structural validation and rejects the payload if any score is
`NAN`. Mirrors the existing skiplist-format check. `inf`/`-inf` and
large finite scores remain accepted (only `NAN` is rejected), matching
normal `ZADD` semantics.

## Testing

- `tests/integration/corrupt-dump.tcl`: a `RESTORE`-path test asserting
rejection and that the server stays up.
- Verified the test **fails on the pre-fix code** (server crashes on
conversion) and **passes after the fix**, by stashing the fix during the
run.
- Confirmed valid zsets, including `inf`/`-inf`/large finite scores,
still load and convert correctly.
- Full `integration/corrupt-dump` suite: 74 passed, 0 failed.

> [!NOTE]
> Found via structure-aware fuzzing of the RESTORE path. This issue was
generated by AI but verified, with love, by a human.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

When `valkey-cli --rdb` trims the EOF marker, a failed `ftruncate` only
printed a warning but still reported success and exited 0, leaving a
corrupt RDB file. This makes it exit non-zero on that failure, matching
how the `write()` failure is already handled in the same function.

(Thank you, Madelyn, for tag teaming this with me.)

Signed-off-by: Grace <glucier22@gmail.com>

…ey-io#3959)

The Case 3 portion of the test was flaky: after a single round of
`CLUSTER DELSLOTS 0` on R0/R1/R2, the cluster could stay in OK state
and `wait_for_cluster_state fail` would time out with
`Cluster node 1 cluster_state:ok`.

The race is between R0's local DELSLOTS and the gossip already in
flight from R0. After R1 locally clears slot 0, a stale pre-DELSLOTS
packet from R0 (whose myslots still claims slot 0) hits the
isSlotUnclaimed fast path in clusterUpdateSlotsConfigWith and rebinds
slot 0 back to R0 on R1. See:
```
    if (isSlotUnclaimed(j) ||
        server.cluster->slots[j]->configEpoch < senderConfigEpoch ||
        clusterSlotFailoverGranted(j)) {
        ...
        clusterDelSlot(j);
        clusterAddSlot(sender, j);
        ...
    }
```

R0's subsequent "no longer claiming" PINGs cannot undo this, because
that path only sets owner_not_claiming_slot and never clears slots[j]:
```
    if (server.cluster->slots[j] == sender) {
        /* The slot is currently bound to the sender but the sender is no longer
         * claiming it. We don't want to unbind the slot yet as it can cause the cluster
         * to move to FAIL state and also throw client error. Keeping the slot bound to
         * the previous owner will cause a few client side redirects, but won't throw
         * any errors. We will keep track of the uncertainty in ownership to avoid
         * propagating misinformation about this slot's ownership using UPDATE
         * messages. */
        bitmapSetBit(server.cluster->owner_not_claiming_slot, j);
    }
```

Combined with clusterUpdateState's full-coverage check looking only
at slots[j] == NULL, R1 stays at cluster OK forever.
```
    if (server.cluster->slots[j] == NULL || ...) {
        new_state = CLUSTER_FAIL;
        ...
    }
```

Rather than fighting the protocol's intentional asymmetry around
"soft delete" via gossip, just retry the DELSLOTS pass until all
three nodes converge to FAIL. This keeps the test focused on the
CLUSTERSCAN error semantics it actually wants to verify.

This closes valkey-io#3891. The test was added in valkey-io#3674.

Signed-off-by: Binbin <binloveplay1314@qq.com>

## Summary

`addReplyCommandSubCommands` unconditionally called `addReplySetLen(c, 0)`
when a command has no subcommands, emitting a RESP3 Set type prefix (`~0`)
regardless of the `use_map` parameter. The non-empty path (below it) already
branches correctly on `use_map` — the empty early-return was simply missing
the same logic.

In RESP3, `COMMAND INFO <cmd>` returns the subcommands field as a Set (`~0`)
instead of an Array (`*0`) for any command without subcommands (e.g. PING).
Strict RESP3 client libraries that dispatch on collection type will
misinterpret the response. Not visible in RESP2 since both Set and Array use
the `*` prefix there.

## Fix

Apply the same `use_map` branch to the empty case:
- `addReplyMapLen(c, 0)` when `use_map=1`
- `addReplyArrayLen(c, 0)` otherwise

## Test

Added a `readraw` integration test in `tests/unit/introspection-2.tcl` that
inspects the raw wire-level type byte for the subcommands field of `COMMAND
INFO ping` in RESP3 mode, asserting `*0` (Array) rather than `~0` (Set).

Signed-off-by: Rick Ramsay <49293857+rickrams@users.noreply.github.com>
Signed-off-by: rickrams <rickrams@amazon.com>

…ey-io#3811)

`off_t` (64-bit), but were read into `int` (32-bit) locals in
`genValkeyInfoString()` and `handleBioThreadFinishedRDBDownload()`.

This causes INFO replication to report negative
`master_sync_total_bytes` during bio disk-based sync when RDB exceeds
2GB.

Fix: change the local variable types from `int` to `off_t`.

Signed-off-by: chx9 <lovelypiska@outlook.com>

…r fuzzy provenance matches (valkey-io#3933)

- update verify-provenance to require near-duplicate evidence for fuzzy
provenance matches
 - normalize master/primary and slave/replica terminology.
 - print the captured provenance script log in the check workflow.

---------

Signed-off-by: Ping Xie <pingxie@outlook.com>

…on (valkey-io#3968)

Adds REUSE 3.3 structure (REUSE.toml, LICENSES/) covering all source
files and vendored dependencies.

Replaces the non-standard dual-license COPYING file with a standard
BSD-3-Clause text.

Updates the description of custom patches to Lua and Jemalloc in
deps/README.md.

Benefits:

- GitHub and OpenSSF Scorecard correctly detect the project license
- reuse spdx generates a complete SPDX SBOM covering first-party code
  and all vendored deps
- CI check prevents future contributors from introducing invalid SPDX
  identifiers
- Per-file license clarity for downstream consumers (distro packagers,
  enterprises)

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…ey-io#3803)

New hashtable API hashtableScanHasPassedKey.  New guarantees around hashtable scan.

---------

Signed-off-by: Rain Valentine <rainval@amazon.com>

Updated `kvstoreScan` to more clearly show when a kvstore cursor was used vs. a hashtable cursor.

Signed-off-by: Jim Brunner <brunnerj@amazon.com>

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>