Optimistic locking for string objects - compare-and-set and compare-and-delete by minchopaskal · Pull Request #14435 · redis/redis

minchopaskal · 2025-10-16T13:57:45Z

Description

Add optimistic locking for string objects via compare-and-set and compare-and-delete mechanism.

What's changed

Introduction of new DIGEST command for string objects calculated via XXH3 hash.

Extend SET command with new parameters supporting optimistic locking. The new value is set only if checks against a given (old) value or a given string digest pass.

Introduction of new DELEX command to support conditionally deleting a key. Conditions are also checks against string value or string digest.

Motivation

For developers who need to to implement a compare-and-set and compare-and-delete single-key optimistic concurrency control this PR provides single-command based implementation.

Compare-and-set and compare-and-delete are mostly used for Optimistic concurrency control: a client (1) fetches the value, keeps the old value (or its digest, for a large string) in memory, (2) manipulates a local copy of the value, (3) applies the local changes to the server, but only if the server’s value hasn’t been changed (still equal to the old value).

Note that compare-and-set can also be implemented with WATCH … MULTI … EXEC and Lua scripts. The new SET optional arguments and the DELEX command do not enable new functionality, however, they are much simpler and faster to use for the very common use case of single-key optimistic concurrency control.

Related issues and PRs

#12485
#8361
#4258

Description of the new commands

DIGEST

DIGEST key

Get the hash digest of the value stored in key, as an hex string.

Reply:

Null if key does not exist
error if key exists but holds a value which is not a string
(bulk string) the XXH3 digest of the value stored in key, as an hex string

SET

SET key value [NX | XX | IFEQ match-value | IFNE match-value | IFDEQ match-digest | IFDNE match-digest] [GET] [EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds | KEEPTTL]

IFEQ match-value - Set the key’s value and expiration only if its current value is equal to match-value. If key doesn’t exist - it won’t be created.
IFNE match-value - Set the key’s value and expiration only if its current value is not equal to match-value. If key doesn’t exist - it will be created.
IFDEQ match-digest - Set the key’s value and expiration only if the digest of its current value is equal to match-digest. If key doesn’t exist - it won’t be created.
IFDNE match-digest - Set the key’s value and expiration only if the digest of its current value is not equal to match-digest. If key doesn’t exist - it will be created.

Reply update:

If GET was not specified:
- Nil reply if either
  - the key doesn’t exist and XX/IFEQ/IFDEQ was specified. The key was not created.
  - the key exists, and NX was specified or a specified IFEQ/IFNE/IFDEQ/IFDNE condition is false. The key was not set.
- Simple string reply: OK: The key was set.
If GET was specified, any of the following:
- Nil reply: The key didn't exist before this command (whether the key was created or not).
- Bulk string reply: The previous value of the key (whether the key was set or not).

DELEX

DELEX key [IFEQ match-value | IFNE match-value | IFDEQ match-digest | IFDNE match-digest]

Conditionally removes the specified key. A key is ignored if it does not exist.

IFEQ match-value - Delete the key only if its value is equal to match-value
IFNE match-value - Delete the key only if its value is not equal to match-value
IFDEQ match-digest - Delete the key only if the digest of its value is equal to match-digest
IFDNE match-digest - Delete the key only if the digest of its value is not equal to match-digest

Reply:

error if key exists but holds a value that is not a string and IFEQ/IFNE/IFDEQ/IFDNE is specified.
(integer) 0 if not deleted (the key does not exist or a specified IFEQ/IFNE/IFDEQ/IFDNE condition is false), or 1 if deleted.

Notes

Added copy of xxhash repo to deps - version

Take care of async del in DELEX

src/db.c

sggeorgiev · 2025-10-17T05:18:19Z

Could you please clarify and document in the PR description how the new conditional operations handle key expiration scenarios? Specifically:

Race condition with expiration: What is the expected behavior when a key expires between the initial read (getting the value/digest) and the conditional SET operation?
IFNE with expired keys: How does SET key value IFNE match-value behave when the key has just expired? Since IFNE creates the key if it doesn't exist, does an expired key count as "doesn't exist" for this purpose?
DELEX with expiration: Similarly, for DELEX key IFEQ match-value, what happens if the key expires between checking the condition and performing the deletion?
Digest validation on expired keys: Does IFDEQ/IFDNE treat an expired key as having no digest (operation fails) or as non-existent (follows the create/no-create logic)?

src/server.h

src/t_string.c

src/commands/set.json

src/t_string.c

src/db.c

LiorKogan · 2025-10-18T06:13:37Z

Re #14435 (comment)

A key could be deleted, renamed, evicted, or expired between the read and the conditional SET. We should not implement any special logic for these use cases. To the best of my knowledge, we don't do that in other places.

For SET, a deleted key is equivalent to not-equal, because, basically, it was changed by another user. An evicted or expired key - well, it is the responsibility of the user to ensure it won't happen.

src/t_string.c

Co-authored-by: debing.sun <debing.sun@redis.com>

src/db.c

src/t_string.c

Co-authored-by: debing.sun <debing.sun@redis.com>

ShooterIT · 2025-10-20T12:04:08Z

Introduction of new DIGEST command for string objects calculated via XXH3 hash.

why do we use XXH3 hash?

LiorKogan · 2025-10-20T12:20:32Z

why do we use XXH3 hash?

It is 64-bit, which, I believe, is a good tradeoff for our use case.

It has good properties. Specifically, speed and quality. See table here.

It is widely adopted.

ShooterIT

great job, developers in the community have wanted this feature for too long, thanks

src/db.c

sundb · 2025-10-20T11:39:59Z

src/commands/delex.json

+        "key_specs": [
+            {
+                "flags": [
+                    "RW",


Suggested change

"RW",

"RM",

I'm not sure that's right.
This mutually exclusive flag defines what Redis physically does to the key (the other one is a set of logical operations from the user's perspective).
Considering that, this command is similar to GETDEL. e.g. for ROF, it'll need to load the data to compare it (it doesn't rely on that, but it's an indication this could be wrong).
@guybe7 please correct me if I'm wrong.

WDYM it's similar to GETDEL? It does not return the old value. Maybe you mean we need the ACCESS flag?

no, i mean there's one set of (mutually exclusive) flags that define how redis uses the data in the key, in that set this command is similar to GETDEL since redis does need the data in the key, so it'w RM.

the other set of (non-mutually exclusive) flags, is the user's perspective, and the user doesn't ACCESS the data (note that we've documented that side-channel, like the user being able to observe the content of the key via DELEX doesn't count as ACCESS)

/* The following refer what the command actually does with the value or metadata * of the key, and not necessarily the user data or how it affects it. * Each key-spec may must have exactly one of these. Any operation that's not * distinctly deletion, overwrite or read-only would be marked as RW. */ #define CMD_KEY_RO (1ULL<<0) /* Read-Only - Reads the value of the key, but * doesn't necessarily returns it. */ #define CMD_KEY_RW (1ULL<<1) /* Read-Write - Modifies the data stored in the * value of the key or its metadata. */ #define CMD_KEY_OW (1ULL<<2) /* Overwrite - Overwrites the data stored in * the value of the key. */ #define CMD_KEY_RM (1ULL<<3) /* Deletes the key. */ /* The following refer to user data inside the value of the key, not the metadata * like LRU, type, cardinality. It refers to the logical operation on the user's * data (actual input strings / TTL), being used / returned / copied / changed, * It doesn't refer to modification or returning of metadata (like type, count, * presence of data). Any write that's not INSERT or DELETE, would be an UPDATE. * Each key-spec may have one of the writes with or without access, or none: */ #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from * the value of the key. */ #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may * depend on the old value. */ #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of * modification or deletion of existing data. */ #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content * from the value of the key. */

I see GETDEL uses RW instead of RM but I'm not sure why exactly. For DELEX the key is deleted, conditionally but still deleted, so not sure why RM is not the correct one for DELEX (and GETDEL for that matter).

Also from the ACCESS docs it feels like it might be correct for DELEX as it uses the data. Or maybe I don't get the terminology correctly.

/* The following refer what the command actually does with the value or metadata * of the key, and not necessarily the user data or how it affects it.

this refers to the redis code, does redis access the value?
try thinking of a key that's on flash in ROF, it does need to load the data to ram in order to read it.

Also from the ACCESS docs it feels like it might be correct for DELEX as it uses the data. Or maybe I don't get the terminology correctly.

i think you might have got it reverse, the short ones (RM and RW) are about what redis does with the data, and the long one (ACCESS and DELETE) are the logical operation from the user's perspective

thanks for the clarification.
The redis code does access the data and does modify it (delete).
So RM is specific for delete only cases - which are DEL and UNLINK currently.
I'll open a PR

@oranagra I'm wondering if we can optimize the comments; otherwise, it's very likely to cause confusion.

#define CMD_KEY_RW (1ULL<<1) /* Read-Write - Modifies or Deletes the data stored in the * value of the key or its metadata. */ #define CMD_KEY_RM (1ULL<<3) /* Deletes the key without reading it. */

ok..
maybe also: Deletes the key without reading it's value

What about

Read-Write - Reads and modifies/deletes the data stored in the value of the key or its metadata.

Co-authored-by: Yuan Wang <wangyuancode@163.com>

Co-authored-by: debing.sun <debing.sun@redis.com>

sundb · 2025-10-20T12:44:06Z

@minchopaskal don't forget to generate commands.def again.

sundb

LGTM

src/db.c

Co-authored-by: debing.sun <debing.sun@redis.com>

In #14435 the `RM` flag was incorrect for the DELEX command as the Redis command code accesses and uses the value of the key, it needs to be RW (just like `GETDEL` command).

minchopaskal added 6 commits October 16, 2025 16:47

Introduce digestCommand and xxhash dependency

7cc1b5d

Add DELEX command and tests for it

01cce76

Extend the SET command with IFEQ/IFNE/IFDEQ/IFDNE

937a278

Fix centos and 32-bit build. Fix redis-cli tests

6fb580d

cut unneeded parts of xxhash code from deps

0186c5b

Add missing xxhash license

8d23812

minchopaskal requested review from oranagra, sggeorgiev and skaslev October 16, 2025 13:57

minchopaskal added state:needs-doc-pr requires a PR to redis-doc repository release-notes indication that this issue needs to be mentioned in the release notes labels Oct 16, 2025

Allow DELEX to delete non-string keys if no conditions are specified

9628e4c

Take care of async del in DELEX

sggeorgiev reviewed Oct 17, 2025

View reviewed changes

src/db.c Show resolved Hide resolved

Fix DELEX tests and add more error checking

54e3953

minchopaskal requested a review from sggeorgiev October 17, 2025 08:40

sundb reviewed Oct 17, 2025

View reviewed changes

src/server.h Show resolved Hide resolved

src/t_string.c Outdated Show resolved Hide resolved

sundb reviewed Oct 17, 2025

View reviewed changes

src/commands/set.json Show resolved Hide resolved

minchopaskal added 2 commits October 17, 2025 14:10

Make string digest return hex string instead of integer

b26465b

Output hex string of xxhash value in a protable manner

8c4cb8d

minchopaskal force-pushed the cas-n-cad branch from 0c5c125 to 8c4cb8d Compare October 17, 2025 11:29

sundb reviewed Oct 18, 2025

View reviewed changes

src/t_string.c Outdated Show resolved Hide resolved

src/db.c Show resolved Hide resolved

sundb reviewed Oct 18, 2025

View reviewed changes

src/t_string.c Outdated Show resolved Hide resolved

minchopaskal and others added 2 commits October 20, 2025 10:36

Update src/t_string.c

1eefcf1

Co-authored-by: debing.sun <debing.sun@redis.com>

Update src/t_string.c

4db26a0

Co-authored-by: debing.sun <debing.sun@redis.com>

sundb reviewed Oct 20, 2025

View reviewed changes

src/db.c Show resolved Hide resolved

src/t_string.c Show resolved Hide resolved

src/t_string.c Outdated Show resolved Hide resolved

src/t_string.c Show resolved Hide resolved

minchopaskal and others added 3 commits October 20, 2025 13:43

Update src/t_string.c

466e7d3

Co-authored-by: debing.sun <debing.sun@redis.com>

Update src/t_string.c

f77005f

Co-authored-by: debing.sun <debing.sun@redis.com>

Add comments for DELEX command

eb92214

Add comment for DIGEST command

8961878

ShooterIT reviewed Oct 20, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

sundb reviewed Oct 20, 2025

View reviewed changes

minchopaskal and others added 2 commits October 20, 2025 15:40

Update src/db.c

8621f5f

Co-authored-by: Yuan Wang <wangyuancode@163.com>

Update src/commands/delex.json

2144d86

Co-authored-by: debing.sun <debing.sun@redis.com>

regen commands.def

615083b

sundb approved these changes Oct 21, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

Update src/db.c

c2480df

Co-authored-by: debing.sun <debing.sun@redis.com>

minchopaskal merged commit aed879a into redis:unstable Oct 21, 2025
21 checks passed

sundb added this to Redis 8.4 Oct 23, 2025

github-project-automation bot moved this to Todo in Redis 8.4 Oct 23, 2025

sundb moved this from Todo to Done in Redis 8.4 Oct 23, 2025

minchopaskal mentioned this pull request Oct 23, 2025

Fix flags of DELEX command #14452

Merged

oranagra pushed a commit that referenced this pull request Oct 24, 2025

Fix flags of DELEX command (#14452)

078019c

In #14435 the `RM` flag was incorrect for the DELEX command as the Redis command code accesses and uses the value of the key, it needs to be RW (just like `GETDEL` command).

This was referenced Oct 28, 2025

[NEW] GET/SET CAS FLAG #12485

Closed

add SETCAS command #8361

Closed

Using check-and-set (CAS) #5917

Closed

Add SETNE #4258

Closed

michael-grunder mentioned this pull request Oct 28, 2025

Implement DIGEST, DELEX, and new Redis SET options. phpredis/phpredis#2739

Open

oranagra mentioned this pull request Nov 12, 2025

Add variable key-spec flags to SET IF* and DELEX #14529

Merged

Conversation

minchopaskal commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What's changed

Motivation

Related issues and PRs

Description of the new commands

DIGEST

SET

DELEX

Notes

Uh oh!

Uh oh!

sggeorgiev commented Oct 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiorKogan commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShooterIT commented Oct 20, 2025

Uh oh!

LiorKogan commented Oct 20, 2025

Uh oh!

ShooterIT left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

minchopaskal Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sundb commented Oct 20, 2025

Uh oh!

sundb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

minchopaskal commented Oct 16, 2025 •

edited

Loading

LiorKogan commented Oct 18, 2025 •

edited

Loading

minchopaskal Oct 23, 2025 •

edited

Loading