-
Notifications
You must be signed in to change notification settings - Fork 24.4k
Add Atomic Slot Migration (ASM) support #14414
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
* Add cluster plugin * Add cluster_asm.c/.h
Dummy implementation of CLUSTER MIGRATION
* the destination sends node id info to the source * fix double replication stream
refactor to use slot range array
cluster asm api
Add basic filtered replication and write pause Changes: Initial implementation for filtered repl stream delivery to destination node Initial write pause Fixes for crashes & code refactor TODO: Write pause time limit Tests
Add partial flush support to SFLUSH command
- DEBUG ASM-FAILPOINT to set fail point - Error handler for both source and destination side & tcl tests - Error info for task - Destination node sends ACK in Cron - Source node checks applied offset gap and pauses writing - Source node sends stream-eof after slot commands stream drained in beforeSleep
refactor and fix language issues Co-authored-by: Yuan Wang <yuan.wang@redis.com>
rename plugin API
Use task ids for ASM tasks Co-authored-by: Yuan Wang <yuan.wang@redis.com>
* dest as replica * add test cases * Fix test cases * rdb channel uses task id to authenticate * revert syncslots-reply * task id check * use task id to find status
don't expire keys for importing clients skip keys that belong migration slots when active expiration hfe expire randomkey command keys command scan command eviction
Use basic types in cluster plugin api
- slot dict size hint - proxy importing data stream
- flush-like command will cancel slot migration task - switching to replica from master will cancel slot migration task - resolve the source node when starting importing tasks - hide rdb/main channel client in info replication - in client list, migrating client is marked as m, importing client is marked as i - show asm_migrating_buffer and asm_importing_buffer in info memory - reject cluster setslot if there is an active slot migration, and cancel tasks if slot topology is changed
Some improvements and utility functions
- CLUSTER FORGET command - CLUSTER ADDSLOTS/DELSLOTS command - CLUSTER SETSLOTS command - CLUSTER FAILOVER command - CLIENT PAUSE command - server shutdown (SHUTDOWN command or kill)
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
background trim
Cluster asm rebase
Trim improvements and better logs
Test fixes
add cluster prefix to conf and info fields
no clusterCommonInit
Automated performance analysis summaryThis comment was automatically generated given there is performance data available. Using platform named: x86-aws-m7i.metal-24xl for both baseline and comparison. Using triggering environment: ci for both baseline and comparison. In summary:
You can check a comparison in detail via the grafana link Comparison between unstable and cluster-asm.Time Period from 5 months ago. (environment used: oss-standalone) By GROUP change csv:command_group,min_change,q1_change,median_change,q3_change,max_change By COMMAND change csv:command,min_change,q1_change,median_change,q3_change,max_change
Improvements test regexp names: memtier_benchmark-1Mkeys-generic-scan-cursor-count-500-pipeline-10|memtier_benchmark-1Mkeys-generic-scan-cursor-count-5000-pipeline-10 Full Results table:
WARNING: There were 59 benchmarks with NO datapoints for both baseline and comparison. NO datapoints for both baseline and comparison:NO DATAPOINTS test regexp names: defaults|memtier_benchmark-1Mkeys-10B-psetex-expire-use-case|memtier_benchmark-1Mkeys-10B-setex-expire-use-case|memtier_benchmark-1Mkeys-load-string-with-100B-values|memtier_benchmark-1Mkeys-load-string-with-100B-values-pipeline-10|memtier_benchmark-1Mkeys-load-string-with-10B-values|memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-10|memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-100|memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-50|memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-500|memtier_benchmark-1Mkeys-load-string-with-1KiB-values|memtier_benchmark-1Mkeys-load-string-with-1KiB-values-pipeline-10|memtier_benchmark-1Mkeys-load-string-with-20KiB-values|memtier_benchmark-1Mkeys-string-get-100B|memtier_benchmark-1Mkeys-string-get-100B-pipeline-10|memtier_benchmark-1Mkeys-string-get-10B|memtier_benchmark-1Mkeys-string-get-10B-pipeline-10|memtier_benchmark-1Mkeys-string-get-10B-pipeline-100|memtier_benchmark-1Mkeys-string-get-10B-pipeline-50|memtier_benchmark-1Mkeys-string-get-10B-pipeline-500|memtier_benchmark-1Mkeys-string-get-1KiB|memtier_benchmark-1Mkeys-string-get-1KiB-pipeline-10|memtier_benchmark-1Mkeys-string-get-32B|memtier_benchmark-1Mkeys-string-get-32B-pipeline-10|memtier_benchmark-1Mkeys-string-mixed-50-50-set-get-100B-expire|memtier_benchmark-1Mkeys-string-mixed-50-50-set-get-100B-expire-pipeline-10|memtier_benchmark-1Mkeys-string-set-with-ex-100B-pipeline-10|memtier_benchmark-1Mkeys-string-setget200c-1KiB-pipeline-1|memtier_benchmark-1key-zset-1M-elements-zremrangebyscore-pipeline-10|memtier_benchmark-3Mkeys-load-string-with-512B-values|memtier_benchmark-3Mkeys-load-string-with-512B-values-pipeline-10|memtier_benchmark-3Mkeys-string-get-with-1KiB-values-400_conns|memtier_benchmark-3Mkeys-string-get-with-1KiB-values-40_conns|memtier_benchmark-3Mkeys-string-get-with-1KiB-values-pipeline-10-2000_conns|memtier_benchmark-3Mkeys-string-get-with-1KiB-values-pipeline-10-400_conns|memtier_benchmark-3Mkeys-string-get-with-1KiB-values-pipeline-10-40_conns|memtier_benchmark-3Mkeys-string-mixed-50-50-with-512B-values-with-expiration-pipeline-10-400_conns|memtier_benchmark-multiple-hll-pfcount-100B-values|memtier_benchmark-multiple-hll-pfmerge-100B-values|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-100-subscribers|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-1000-subscribers|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-5000-subscribers|memtier_benchmark-nokeys-pubsub-mixed-100-channels-128B-100-publishers-50K-subscribers-5k-conns|memtier_benchmark-nokeys-server-time-pipeline-10|memtier_benchmark-playbook-leaderboard-top-10|memtier_benchmark-playbook-leaderboard-top-100|memtier_benchmark-playbook-leaderboard-top-1000|memtier_benchmark-playbook-rate-limiting-lua-100k-sessions|memtier_benchmark-playbook-realtime-analytics-membership|memtier_benchmark-playbook-realtime-analytics-membership-pipeline-10|memtier_benchmark-playbook-session-caching-hash-100k-sessions|memtier_benchmark-playbook-session-caching-json-100k-sessions|memtier_benchmark-playbook-session-caching-string-100k-sessions|memtier_benchmark-playbook-session-storage-100k-sessions|memtier_benchmark-playbook-session-storage-1k-sessions|memtier_benchmark-stream-10M-entries-xread-count-100|memtier_benchmark-stream-10M-entries-xreadgroup-count-100|memtier_benchmark-stream-10M-entries-xreadgroup-count-100-noack|memtier_benchmark-stream-concurrent-xadd-xreadgroup-70-30 |
ShooterIT
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
exciting work, LGTM :)
|
@ShooterIT Exciting work from you!!!! You took on the most complex parts and nailed them. It was great (and fun) working on this project together :) |
Overview
This PR is a joint effort with @ShooterIT . I’m just opening it on behalf of both of us.
This PR introduces Atomic Slot Migration (ASM) for Redis Cluster — a new mechanism for safely and efficiently migrating hash slots between nodes.
Redis Cluster distributes data across nodes using 16384 hash slots, each owned by a specific node. Sometimes slots need to be moved — for example, to rebalance after adding or removing nodes, or to mitigate a hot shard that’s overloaded. Before ASM, slot migration was non-atomic and client-dependent, relying on CLUSTER SETSLOT, GETKEYSINSLOT, MIGRATE commands, and client-side handling of ASK/ASKING replies. This process was complex, error-prone, slow and could leave clusters in inconsistent states after failures. Clients had to implement redirect logic, multi-key commands could fail mid-migration, and errors often resulted in orphaned keys or required manual cleanup. Several related discussions can be found in the issue list, some examples: #14300 , #4937 , #10370 , #4333 , #13122, #11312
Atomic Slot Migration (ASM) makes slot rebalancing safe, transparent, and reliable, addressing many of the limitations of the legacy migration method. Instead of moving keys one by one, ASM replicates the entire slot’s data plus live updates to the target node, then performs a single atomic handoff. Clients keep working without handling ASK/ASKING replies, multi-key operations remain consistent, failures don’t leave partial states, and replicas stay in sync. The migration process also completes significantly faster. Operators gain new commands (CLUSTER MIGRATION IMPORT, STATUS, CANCEL) for monitoring and control, while modules can hook into migration events for deeper integration.
The problems of legacy method in detail
Operators and developers ran into multiple issues with the legacy method, some of these issues in detail:
-ASKand-ASKINGresponses, reissuing requests to the target node. Not all client libraries implemented this correctly, leading to failed commands or subtle bugs. Even when implemented, it increased latency and broke naive pipelines.MGET key1 key2could fail withTRYAGAINif part of the slot was already migrated. This made application logic unpredictable during resharding.How Atomic Slot Migration Fixes This
Atomic Slot Migration (ASM) eliminates all of these issues by:
ASM Diagram and Migration Steps
New commands introduced
There are two new commands:
CLUSTER MIGRATION <arg>CLUSTER SYNCSLOTS <arg>For more details, please refer to the New Commands section. Internal command messaging is mostly omitted in the diagram for simplicity.Steps
Slot migration begins when the operator sends
CLUSTER MIGRATION IMPORT <start-slot> <end-slot> ...to the destination master. The process is initiated from the destination node, similar to REPLICAOF. This approach allows us to reuse the same logic and share code with the new replication mechanism (see Rdb channel replication #13732). The command can include multiple slot ranges. The destination node creates one migration task per source node, regardless of how many slot ranges are specified. Upon successfully creating the task, the destination node replies IMPORT command with the assigned task ID. The operator can then monitor progress using
CLUSTER MIGRATION STATUS ID <task-id>. When the task’s state field changes tocompleted, the migration has finished successfully. Please see New Commands section for the output sample.After creating the migration task, the destination node will request replication of slots by using the internal command
CLUSTER SYNCSLOTS.Once the source node accepts the request, the destination node establishes another separate connection(similar to rdbchannel replication) so snapshot data and incremental changes can be transmitted in parallel.
Source node forks, starts delivering snapshot content (as per-key RESTORE commands) from one connection and incremental changes from the other connection. The destination master starts applying commands from the snapshot connection and accumulates incremental changes. Applied commands are also propagated to the destination replicas via replication backlog.
Note: Only commands of related slots are delivered to the destination node. This is done by writing them to the migration client’s output buffer, which serves as the replication stream for the migration operation.
Once the source node finishes delivering the snapshot and determines that the destination node has caught up (remaining repl stream to consume went under a configured limit), it pauses write traffic for the entire server. After pausing the writes, the source node forwards any remaining write commands to the destination node.
Once the destination consumes all the writes, it bumps up cluster config epoch and changes the configuration. New config is published via cluster bus.
When the source node receives the new configuration, it can redirect clients and it begins trimming the migrated slots, while also resuming write traffic on the server.
Internal slots synchronization state machine
CLUSTER SYNCSLOTS SYNC <task-id> <start-slot> <end-slot>to initiate a slot synchronization request and establish the main channel. The source node responds with+RDBCHANNELSYNCSLOTS, indicating that the destination node should establish an RDB channel.CLUSTER SYNCSLOTS RDBCHANNEL <task-id>to establish the RDB channel, using the same task-id as in the previous step to associate the two connections as part of the same ASM task.The source node replies with
+SLOTSSNAPSHOT, andforka child process to transfer slot snapshot.CLUSTER SYNCSLOTS SNAPSHOT-EOFcommand. The destination node then starts streaming the buffered commands while continuing to read and buffer incremental commands sent from the source.CLUSTER SYNCSLOTS ACK <offset>to inform the source of the applied data offset. When the offset gap meets the threshold, the source node pauses write operations. After all buffered data has been drained, it sendsCLUSTER SYNCSLOTS STREAM-EOFto the destination node to hand off slots.Error handling
Slot Snapshot Format Considerations
When the source node forks to deliver slot content, in theory, there are several possible formats for transmitting the snapshot data:
Mini RDB:A compact RDB file containing only the keys from the migrating slots. This format is efficient for transmission, but it cannot be easily forwarded to destination-side replicas.
AOF format: The source node can generate commands in AOF form (e.g., SET x y, HSET h f v) and stream them. Individual commands are easily appended to the replication stream and propagated to replicas. Large keys can also be split into multiple commands (incrementally reconstructing the value), similar to the AOF rewrite process.
RESTORE commands: Each key is serialized and sent as a
RESTOREcommand. These can be appended directly to the destination’s replication stream, though very large keys may make serialization and transmission less efficient.We chose the
RESTOREcommand as default approach for the following reasons:Some details:
Trimming the keys
When a migration completes successfully, the source node deletes the migrated keys from its local database.
Since the migrated slots may contain a large number of keys, this trimming process must be efficient and non-blocking.
In cluster mode, Redis maintains per-slot data structures for keys, expires, and subexpires. This organization makes it possible to efficiently detach all data associated with a given slot in a single step. During trimming, these slot-specific data structures are handed off to a background I/O (BIO) thread for asynchronous cleanup—similar to how FLUSHALL or FLUSHDB operate. This mechanism is referred to as background trimming, and it is the preferred and default method for ASM, ensuring that the main thread remains unblocked.
However, unlike Redis itself, some modules may not maintain per-slot data structures and therefore cannot drop related slots data in a single operation. To support these cases, Redis introduces active trimming, where key deletion occurs in the main thread instead. This is not a blocking operation, trimming runs concurrently in the main thread, periodically removing keys during the cron loop. Each deletion triggers a keyspace notification so that modules can react to individual key removals. While active trim is less efficient, it ensures backward compatibility for modules during the transition period.
Before starting the trim, Redis checks whether any module is subscribed to newly added
REDISMODULE_NOTIFY_KEY_TRIMMEDkeyspace event. If such subscribers exist, active trimming is used; otherwise, background trimming is triggered. Going forward, modules are expected to adopt background trimming to take advantage of its performance and scalability benefits, and active trimming will be phased out once modules migrate to the new model.Redis also prefers active trimming if there is any client that is using client tracking feature (see client-side caching). In the current client tracking protocol, when a database is flushed (e.g., via the FLUSHDB command), a null value is sent to tracking clients to indicate that they should invalidate all locally cached keys. However, there is currently no mechanism to signal that only specific slots have been flushed. Iterating over all keys in the slots to be trimmed would be a blocking operation. To avoid this, if there is any client that is using client tracking feature, Redis automatically switches to active trimming mode. In the future, the client tracking protocol can be extended to support slot-based invalidation, allowing background trimming to be used in these cases as well.
Finally, trimming may also be triggered after a migration failure. In such cases, the operation ensures that any partially imported or inconsistent slot data is cleaned up, maintaining cluster consistency and preventing stale keys from remaining in the source or destination nodes.
Note about active trim: Subsequent migrations can complete while a prior trim is still running. In that case, the new migration’s trim job is queued and will start automatically after the current trim finishes. This does not affect slot ownership or client traffic—it only serializes the background cleanup.
Replica handling
During importing, new keys are propagated to destination side replica. Replica will check slot ownership before replying commands like SCAN, KEYS, DBSIZE not to include these unowned keys in the reply.
Also, when an import operation begins, the master now propagates an internal command through the replication stream, allowing replicas to recognize that an ASM operation is in progress. This is done by the internal
CLUSTER SYNCSLOTS CONF ASM-TASKcommand in the replication stream. This enables replicas to trigger the relevant module events so that modules can adapt their behavior — for example, filtering out unowned keys from read-only requests during ASM operations. To be able to support full sync with RDB delivery scenarios, a new AUX field is also added to the RDB:cluster-asm-task. It's value is a string in the format oftask_id:source_node:dest_node:operation:state:slot_ranges.After a successful migration or on a failed import, master will trim the keys. In that case, master will propagate a new command to the replica:
TRIMSLOTS RANGES <numranges> <start-slot> <end-slot> .... So, the replica will start trimming once this command is received.Propagating data outside the keyspace
When the destination node is newly added to the cluster, certain data outside the keyspace may need to be propagated first.
A common example is functions. Previously, redis-cli handled this by transferring functions when a new node was added.
With ASM, Redis now automatically dumps and sends functions to the destination node using
FUNCTION RESTORE ..REPLACEcommand — done purely for convenience to simplify setup.Additionally, modules may also need to propagate their own data outside the keyspace.
To support this, a new API has been introduced:
RM_ClusterPropagateForSlotMigration().See the Module Support section for implementation details.
Limitations
Single migration at a time: Only one ASM migration operation is allowed at a time. This limitation simplifies the current design but can be extended in the future.
Large key handling: For large keys, ASM switches to AOF encoding to deliver key data in chunks. This mechanism currently applies only to non-module keys. In the future, the RESTORE command may be extended to support chunked delivery, providing a unified solution for all key types. See Slot Snapshot Format Considerations for details.
There are several cases that may cause an Atomic Slot Migration (ASM) to be aborted (can be retried afterwards):
When active trimming is enabled, a node must not re-import the same slots while trimming for those slots is still in progress. Otherwise, it can’t distinguish newly imported keys from pre-existing ones, and the trim cron might delete the incoming keys by mistake. In this state, the node rejects IMPORT operation for those slots until trimming completes. If the master has finished trimming but a replica is still trimming, master may still start the import operation for those slots. So, the replica checks whether the master is sending commands for those slots; if so, it blocks the master’s client connection until trimming finishes. This is a corner case, but we believe the behavior is reasonable for now. In the worst case, the master may drop the replica (e.g., buffer overrun), triggering a new full sync.
API Changes
New Commands
Public commands
Syntax:
CLUSTER MIGRATION IMPORT <start-slot> <end-slot> [<start-slot> <end-slot>]...Args: Slot ranges
Reply:
Description: Executes on the destination master. Accepts multiple slot ranges and triggers atomic migration for the specified ranges. Returns a task ID that can be used to monitor the status of the task. In CLUSTER MIGRATION STATUS output, “state” field will be
completedon a successful operation.Syntax:
CLUSTER MIGRATION CANCEL [ID <id> | ALL]Args: Task ID or ALL
Reply: Number of cancelled tasks
Description: Cancels an ongoing migration task by its ID or cancels all tasks if ALL is specified. Note: Cancelling a task on the source node does not stop the migration on the destination node, which will continue retrying until it is also cancelled there.
Syntax:
CLUSTER MIGRATION STATUS [ID <id> | ALL]Args: Task ID or ALL
Reply:
Description: Displays the status of all current and completed atomic slot migration tasks. If a specific task ID is provided, it returns detailed information for that task only.
"import"or"migrate"."completed"when finished.Sample output:
Internal commands
Syntax:
CLUSTER SYNCSLOTS <arg> ...Args: Internal messaging operations
Reply: +OK or -ERR on failure (e.g. invalid slot range)
Description: Used for internal communication between source and destination nodes. e.g. handshaking, establishing multiple channels, triggering handoff.
Syntax:
TRIMSLOTS RANGES <numranges> <start-slot> <end-slot> ...Args: Slot ranges to trim
Reply: +OK
Description: Master propagates it to replica so that replica can trim unowned keys after a successful migration or on a failed import.
New configs
cluster-slot-migration-max-archived-tasks: To list inCLUSTER MIGRATION STATUS ALLoutput, Redis keeps last n migration tasks in memory. This config controls maximum number of archived ASM tasks. Default value: 32, used as a hidden configcluster-slot-migration-handoff-max-lag-bytes: After the slot snapshot is completed, if the remaining replication stream size falls below this threshold, the source node pauses writes to hand off slot ownership. A higher value may trigger the handoff earlier but can lead to a longer write pause, since more data remains to be replicated. A lower value can result in a shorter write pause, but it may be harder to reach the threshold if there is a steady flow of incoming writes. Default value: 1MBcluster-slot-migration-write-pause-timeout: The maximum duration (in milliseconds) that the source node pauses writes during ASM handoff. After pausing writes, if the destination node fails to take over the slots within this timeout (for example, due to a cluster configuration update failure), the source node assumes the migration has failed and resumes writes to prevent indefinite blocking. Default value: 10 secondscluster-slot-migration-sync-buffer-drain-timeout: Timeout in milliseconds for sync buffer to be drained during ASM.After the destination applies the accumulated buffer, the source continues sending commands for migrating slots. The destination keeps applying them, but if the gap remains above the acceptable limit (see
slot-migration-handoff-max-lag-bytes), which may cause endless synchronization. A timeout check is required to handle this case.The timeout is calculated as the maximum of two values:
Default value: 60000 millliseconds, used as a hidden config
New flag in CLIENT LIST
oflag.gflag.New INFO fields
mem_cluster_slot_migration_output_buffer: Memory usage of the migration client’s output buffer. Redis writes incoming changes to this buffer during the migration process.mem_cluster_slot_migration_input_buffer: Memory usage of the accumulated replication stream buffer on the importing node.mem_cluster_slot_migration_input_buffer_peak: Peak accumulated repl buffer size on the importing sideNew CLUSTER INFO fields
cluster_slot_migration_active_tasks: Number of in-progress ASM tasks. Currently, it will be 1 or 0.cluster_slot_migration_active_trim_running: Number of active trim jobs in progress and scheduledcluster_slot_migration_active_trim_current_job_keys: Number of keys scheduled for deletion in the current trim job.cluster_slot_migration_active_trim_current_job_trimmed: Number of keys already deleted in the current trim job.cluster_slot_migration_stats_active_trim_started: Total number of trim jobs that have started since the process began.cluster_slot_migration_stats_active_trim_completed: Total number of trim jobs completed since the process began.cluster_slot_migration_stats_active_trim_cancelled: Total number of trim jobs cancelled since the process began.Changes in RDB format
A new aux field is added to RDB:
cluster-asm-task. When an import operation begins, the master now propagates an internal command through the replication stream, allowing replicas to recognize that an ASM operation is in progress. This enables replicas to trigger the relevant module events so that modules can adapt their behavior — for example, filtering out unowned keys from read-only requests during ASM operations. To be able to support RDB delivery scenarios, a new field is added to the RDB. See replica handlingBug fix
We don't plan to backport these fixes into old versions, since these are very rare cases.
Keys visibility
When performing atomic slot migration, during key importing on the destination node or key trimming on the source/destination, these keys will be filtered out in the following commands:
The only command that will reflect the increasing number of keys is:
Module Support
NOTE: Please read trimming section and see how does ASM decide about trimming method when there are modules in use.
New notification:
When a key is deleted by the active trim operation, this notification will be sent to subscribed modules.
Also, ASM will automatically choose the trimming method depending on whether there are any subscribers to this new event. Please see the further details here: trimming
New struct in the API:
New Events
1. REDISMODULE_EVENT_CLUSTER_SLOT_MIGRATION (RedisModuleEvent_ClusterSlotMigration)
These events notify modules about different stages of Active Slot Migration (ASM) operations such as when import or migration starts, fails, or completes. Modules can use these notifications to track cluster slot movements or perform custom logic during ASM transitions.
Parameter to these events:
2. REDISMODULE_EVENT_CLUSTER_SLOT_MIGRATION_TRIM (RedisModuleEvent_ClusterSlotMigrationTrim)
These events inform modules about the lifecycle of ASM key trimming operations. Modules can use them to detect when trimming starts, completes, or is performed asynchronously in the background.
Parameter to these events:
New functions
ASM API for alternative cluster implementations
Following #12742, Redis cluster code was restructured to support alternative cluster implementations. Redis uses cluster_legacy.c implementation by default. This PR adds a generic ASM API so alternative implementations can initiate and coordinate Atomic Slot Migration (ASM) while Redis executes the data movement and emits state changes.
Documentation rests in
cluster.h:Co-authored-by: Yuan Wang yuan.wang@redis.com