Skip to content

Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode#2555

Open
enjoy-binbin wants to merge 20 commits into
valkey-io:unstablefrom
enjoy-binbin:bio_conf
Open

Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode#2555
enjoy-binbin wants to merge 20 commits into
valkey-io:unstablefrom
enjoy-binbin:bio_conf

Conversation

@enjoy-binbin

@enjoy-binbin enjoy-binbin commented Aug 27, 2025

Copy link
Copy Markdown
Member

When the cluster changes, we need to persist the cluster configuration.
Currently, nodes.conf is saved synchronously in the main thread during
clusterBeforeSleep, if I/O is delayed or blocked, possibly by disk
contention, this may result in large latencies on the main thread that
affect client requests.

We should avoid synchronous I/O from the main thread.

In this commit, we will try to use bio to save the config file when
cluster-config-save-behavior is set to best-effort. We add a new bio job
and send a sds version of the config file, which does the synchronous save,
so there is some eventually consistent version consistently stored on disk.

For shutdown and cluster saveconfig, we will wait for the bio job to get
drained and trigger a new save in a sync way.

New cluster info fields:

  • cluster_config_save_status: ok or err.
  • cluster_config_last_save_time: the unix time of the last successful save.

New DEBUG BIO-DRAIN subcommand to drain the bio jobs.

Closes #2424.

When the cluster changes, we need to persist the cluster configuration,
if I/O is delayed or blocked, possibly by disk contention, this may result
in large latencies on the main thread.

We should avoid synchronous I/O from the main thread. So in this commit,
we will try to bio to save the config file. We add a bio job and send a
sds version of the config file, which does the synchronous save, so there
is some eventually consistent version consistently stored on disk.

This may break our previous assumption that nodes.conf is in sync and has
the strong consistency. For shutdown and cluster saveconfig, we will wait
for the bio job to get drained and trigger a new save in a sync way.

Closes valkey-io#2424.

Signed-off-by: Binbin <binloveplay1314@qq.com>
@enjoy-binbin enjoy-binbin marked this pull request as ready for review August 27, 2025 12:05
@codecov

codecov Bot commented Aug 27, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.70%. Comparing base (ad3b33e) to head (5940544).
⚠️ Report is 2 commits behind head on unstable.

Additional details and impacted files
@@            Coverage Diff            @@
##           unstable    #2555   +/-   ##
=========================================
  Coverage     76.70%   76.70%           
=========================================
  Files           162      162           
  Lines         80674    80726   +52     
=========================================
+ Hits          61880    61922   +42     
- Misses        18794    18804   +10     
Files with missing lines Coverage Δ
src/bio.c 81.67% <100.00%> (+2.51%) ⬆️
src/cluster_legacy.c 88.60% <100.00%> (+0.21%) ⬆️
src/debug.c 55.15% <100.00%> (+0.19%) ⬆️
src/server.c 89.63% <100.00%> (+0.18%) ⬆️
src/server.h 100.00% <ø> (ø)

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread src/cluster_legacy.c Outdated

@hpatro hpatro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High level thought:

We should make the clusterSaveConfig as async only otherwise we don't have any protection mechanism to disallow stale bio job to overwrite a freshly saved cluster config file from main thread.

Comment thread src/bio.c Outdated
Comment thread src/bio.c Outdated
Comment thread src/bio.c Outdated
Comment thread src/cluster_legacy.c
Comment on lines +929 to +930
if (!from_bio) latencyAddSampleIfNeeded("cluster-config-open", latency);
if (!from_bio) latencyTraceIfNeeded(cluster, cluster_config_open, latency);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any difference? Measuring latency should be the same for main thread/background thread I believe.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latency is not currently thread-safe.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I agree this is not TS we should not prevent the user from being able to relaize he has disk related issues which impacts the config persistance lag. Maybe just a simple cluster_config_last_save_time (same pattern as rdb_last_save_time) which will help users identify when the config save is lagging too long?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, added a new cluster_config_last_save_time cluster info field.

Comment thread src/cluster_legacy.c Outdated
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Comment thread src/bio.c Outdated
bioSubmitJob(BIO_RDB_SAVE, job);
}

void bioCreateClusterConfigSaveJob(sds content, int do_fsync) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who owns/free’s content? bioCreateClusterConfigSaveJob(sds content, int do_fsync) enqueues an sds into job->cluster_save_args.content, then the BIO worker calls clusterSaveConfigFromBio(...). is there a clear free path for content afterwards—Would you please document ownership and ensure a single, deterministic free site to avoid leaks or double-frees.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the dealy, clusterSaveConfigImpl will own and free the content.

Comment thread src/cluster_legacy.c
latencyStartMonitor(latency);
while (offset < content_size) {
written_bytes = write(fd, ci + offset, content_size - offset);
written_bytes = write(fd, content + offset, content_size - offset);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If multiple saves are enqueued, can older content overwrite newer ones and amplify I/O. Can we Coalesce jobs so only the latest content is written (drop older items). Do we need some sort of versioning/epoch so the older jobs are dropped.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, i guess we can do it later?

Comment thread src/bio.c
} else if (job_type == BIO_RDB_SAVE) {
replicaReceiveRDBFromPrimaryToDisk(job->save_to_disk_args.conn, job->save_to_disk_args.is_dual_channel);
} else if (job_type == BIO_CLUSTER_SAVE) {
if (clusterSaveConfigFromBio(job->cluster_save_args.content, job->cluster_save_args.do_fsync) == C_ERR) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to introduce exponential backoff and retry to mitigate transient errors. Can we include observability and metrics around the type of error we notice while trying to save the file/directory

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you want something like aof_last_write_status in the INFO fields?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to introduce exponential backoff and retry to mitigate transient errors.

Cluster configuration file updates should occur infrequently, so i think we are ok?

Can we include observability and metrics around the type of error we notice while trying to save the file/directory

Added a new cluster_config_save_status filed in CLUSTER INFO, does this slove your concern?

Sorry for the delay reply, these related PRs have indeed been dragging on for quite a while, let's pick them back up and merge them into 9.1.

@madolson

Copy link
Copy Markdown
Member

Core team meeting:

  1. Added some reviewers to make sure this makes progress, since this seems to have been forgotten. Some concerns raised about double voting.

@cherukum-Amazon

Copy link
Copy Markdown

Can we target this fix for upcoming patch release?.

@soloestoy

Copy link
Copy Markdown
Member

one concern about shutdown, I think we should call bioDrainWorker in finishShutdown to wait the cluster config write done.

Signed-off-by: Binbin <binloveplay1314@qq.com>
@enjoy-binbin

Copy link
Copy Markdown
Member Author

@cherukum-Amazon sorry for the dealy, i somehow lost the context a while ago, i will try to refresh it this week. Let's start working on #1032 first and try to push it forward.

one concern about shutdown, I think we should call bioDrainWorker in finishShutdown to wait the cluster config write done.

yes, we do call bioDrainWorker in finishShutdown.

@enjoy-binbin enjoy-binbin added the run-extra-tests Run extra tests on this PR (Runs all tests from daily except valgrind and RESP) label Dec 8, 2025
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
@enjoy-binbin enjoy-binbin added the major-decision-pending Major decision pending by TSC team label Dec 30, 2025
@madolson madolson requested a review from rainsupreme January 28, 2026 20:09
@madolson madolson moved this from Todo to Needs Review in Valkey 9.1 Feb 2, 2026

@rainsupreme rainsupreme left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Though, I'm interested in whether it would help to have more detailed error logging or what metrics would actually be helpful

@hpatro hpatro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make a decision on this @valkey-io/core-team. Could you folks vote if we can support cluster config persistence in bio 👍 / 👎 ?

Risks with moving the cluster node configuration persistence to bio (reiterating some of the discussions we had):

  1. lastVoteEpoch is no longer persisted synchronously, potential double voting issue can surface on crash. This issue is similar to not exiting the process on failure to persist the config file.
  2. Put this change behind cluster-persist-config -> best-effort (name to be finalized) config getting introduced in #3372.

I believe operators are fine with offloading this to BIO to avoid dependency on the main thread which can get stuck and the voting correctness issue is of lesser importance.

@zuiderkwast

Copy link
Copy Markdown
Contributor

I thought this was already covered by the major decision we did for #3372. Didn't we discuss these two PRs togeher?

@hpatro

hpatro commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

I thought this was already covered by the major decision we did for #3372. Didn't we discuss these two PRs togeher?

Seems like a miss we didn't mark this PR as major decision approved. Let me update it.

@hpatro hpatro added major-decision-approved Major decision approved by TSC team and removed major-decision-pending Major decision pending by TSC team labels Apr 7, 2026
@lucasyonge

Copy link
Copy Markdown
Contributor

@hpatro I saw you just updated this pr to major-decision-approved, do we need to discuss it in next Monday meeting? (I found it is in the next Monday meeting agenda)

@hpatro

hpatro commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@hpatro I saw you just updated this pr to major-decision-approved, do we need to discuss it in next Monday meeting? (I found it is in the next Monday meeting agenda)

Apparently this was discussed alongside #3372 and was approved over the Monday contributor's meeting. We were just missing the right label. We will wait for binbin to wrap up the changes.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
@enjoy-binbin enjoy-binbin changed the title Save cluster file in bio to avoid the stuck io latency Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode Apr 10, 2026
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
@coderabbitai

coderabbitai Bot commented May 18, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Moves cluster config saves to a BIO background worker. Adds BIO_CLUSTER_SAVE, bioCreateClusterConfigSaveJob(), clusterSaveConfigFromBio(), refactors save logic for BIO/main-thread contexts, tracks atomic save status exposed in CLUSTER INFO, adds DEBUG BIO-DRAIN, and updates tests and docs.

Changes

Cluster Config BIO Save

Layer / File(s) Summary
Job type and function contracts
src/bio.h, src/cluster.h
Define BIO_CLUSTER_SAVE opcode and declare bioCreateClusterConfigSaveJob() and clusterSaveConfigFromBio() contracts.
Atomic save status field
src/server.h, src/server.c
Add _Atomic(int) cluster_config_bio_save_status and _Atomic(time_t) cluster_config_last_save_time to valkeyServer and initialize at startup.
BIO worker routing and job handling
src/bio.c, src/bio.h
Register bio_cluster_config_save worker, route BIO_CLUSTER_SAVE to it, extend bio_job with cluster-save args, add job creation helper, and implement job handler that calls clusterSaveConfigFromBio() with atomic status updates and rate-limited error logging.
Config save implementation with BIO support
src/cluster_legacy.c, src/cluster.h
Add clusterGenNodesConfContent(), refactor core save into clusterSaveConfigImpl(content, from_bio, do_fsync) that owns the SDS payload and gates latency tracing when from BIO, and provide clusterSaveConfigFromBio() and clusterSaveConfigBackground() wrappers.
Config save call site updates
src/cluster_legacy.c
Drain BIO queue before synchronous saves where needed (shutdown, CLUSTER SAVECONFIG), route best-effort saves via background enqueuer in clusterBeforeSleep, and update callers to the boolean do_fsync signature.
Save status exposure in CLUSTER INFO
src/cluster_legacy.c
Atomically read cluster_config_bio_save_status/cluster_config_last_save_time and expose them as cluster_config_save_status ("ok"/"err") and last save time in CLUSTER INFO output.
DEBUG BIO-DRAIN command
src/debug.c
Add DEBUG BIO-DRAIN <type> subcommand to drain background job queues by type for testing.
Test infrastructure and coverage
tests/support/cluster_util.tcl, tests/unit/cluster/misc.tcl
Add get_nodes_conf_path/remove_nodes_conf_folder helpers, enable latency-monitor in cluster tests, and expand best-effort save tests with BIO-drain synchronization and status verification across failure/recovery.
Configuration documentation update
valkey.conf
Expand cluster-config-save-behavior best-effort docs to describe asynchronous BIO saves, warning-only failure handling with retries, and cluster_config_save_status/cluster_config_last_save_time reporting.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 55.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode' clearly and specifically describes the main change in the pull request: using BIO threads for asynchronous cluster config saves in best-effort mode.
Linked Issues check ✅ Passed The PR successfully addresses the requirements of issue #2424 by moving cluster config saves to a BIO thread to eliminate synchronous I/O from the main thread, with drain capability for shutdown/explicit saves.
Out of Scope Changes check ✅ Passed All code changes are directly related to implementing BIO-based cluster config saving, including necessary supporting infrastructure (bio functions, cluster save logic, status tracking, and tests).
Description check ✅ Passed The PR description accurately describes the changeset: async cluster config saving via BIO for best-effort mode, new cluster info fields, and new DEBUG BIO-DRAIN command.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/unit/cluster/misc.tcl (1)

106-110: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Synchronize both nodes before asserting cross-node logs/status.

Line 106 and Line 121 drain only node 0, but subsequent assertions read node -1 state too. In best-effort mode this is racy and can make the test flaky.

Suggested fix
-        R 0 debug bio-drain $BIO_CLUSTER_SAVE
+        R 0 debug bio-drain $BIO_CLUSTER_SAVE
+        R 1 debug bio-drain $BIO_CLUSTER_SAVE
@@
-        R 0 debug bio-drain $BIO_CLUSTER_SAVE
+        R 0 debug bio-drain $BIO_CLUSTER_SAVE
+        R 1 debug bio-drain $BIO_CLUSTER_SAVE

As per coding guidelines "Avoid timing-dependent tests; use proper synchronization (test reliability)".

Also applies to: 121-130

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/cluster/misc.tcl` around lines 106 - 110, The test drains only
node 0 (using the R 0 debug bio-drain $BIO_CLUSTER_SAVE call) but then asserts
logs/status on node -1 with verify_log_message, which is racy; update the test
to synchronize both nodes before cross-node assertions by performing the
equivalent drain/sync on node -1 as well (e.g., invoke the same debug bio-drain
step or a cluster synchronization for node -1) so that both R and
verify_log_message checks operate on deterministic, post-drain state for nodes 0
and -1.
🧹 Nitpick comments (4)
src/bio.h (1)

46-46: ⚡ Quick win

Clarify content ownership in this public BIO API.

bioCreateClusterConfigSaveJob(sds content, bool do_fsync) hands data across threads; please document whether ownership of content is transferred or retained to prevent leaks/UAF at call sites.

As per coding guidelines, "Document why code exists, not just what it does; document all functions in C code".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bio.h` at line 46, bioCreateClusterConfigSaveJob currently hands sds data
across threads but the header prototype lacks ownership semantics; update the
public declaration for bioCreateClusterConfigSaveJob(sds content, bool do_fsync)
in src/bio.h with a clear doc comment that states whether the function takes
ownership of the sds (for example: "Takes ownership of 'content'; caller must
not free it after calling" or "Caller retains ownership; function makes an
internal copy"), describe expected life‑time/thread-transfer behavior and who is
responsible for freeing the sds, and mention any constraints on 'content'
(non-NULL, immutability expectations) and the effect of do_fsync; ensure callers
are audited to match the documented convention.
src/bio.c (1)

334-345: ⚡ Quick win

Reset the log suppressor after a successful save.

last_save_error_log never gets cleared on success, so a fail → recover → fail sequence inside 30 seconds stays silent on the second failure. That hides the transition operators usually care about most. Reset the limiter on success, or gate the warning on the previous cluster_config_bio_save_status as well.

Suggested tweak
         } else if (job_type == BIO_CLUSTER_SAVE) {
             static time_t last_save_error_log = 0;
             if (clusterSaveConfigFromBio(job->cluster_save_args.content, job->cluster_save_args.do_fsync) == C_ERR) {
                 /* Limit logging rate to 1 line per CONFIG_SAVE_LOG_ERROR_RATE seconds. */
                 if ((server.unixtime - last_save_error_log) > CONFIG_SAVE_LOG_ERROR_RATE) {
                     serverLog(LL_WARNING, "Failed to save the cluster config file in background. Cluster config "
                                           "updated even though writing the cluster config file to disk failed.");
                     last_save_error_log = server.unixtime;
                 }
                 atomic_store_explicit(&server.cluster_config_bio_save_status, C_ERR, memory_order_relaxed);
             } else {
                 atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK, memory_order_relaxed);
+                last_save_error_log = 0;
             }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bio.c` around lines 334 - 345, The log rate limiter last_save_error_log
in the BIO_CLUSTER_SAVE branch prevents warnings after a recovery because it’s
never reset; modify the success branch in the BIO_CLUSTER_SAVE handling (the
else after clusterSaveConfigFromBio(...)) to clear or reset last_save_error_log
(e.g. set it to 0 or to server.unixtime - CONFIG_SAVE_LOG_ERROR_RATE) so a
subsequent failure will log again, and ensure
atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK,
memory_order_relaxed) remains unchanged; alternatively, gate the warning on the
previous value of server.cluster_config_bio_save_status to only suppress
repeated fails without masking recoveries.
src/cluster_legacy.c (2)

1039-1047: ⚡ Quick win

Make this helper file-local.

clusterGenNodesConfContent() is only used inside src/cluster_legacy.c, so exporting the symbol unnecessarily widens the surface area.

♻️ Proposed fix
-sds clusterGenNodesConfContent(void) {
+static sds clusterGenNodesConfContent(void) {
As per coding guidelines, `src/**/*.{c,h}`: `Use static keyword for file-local functions in C code`
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` around lines 1039 - 1047, The function
clusterGenNodesConfContent is used only within this translation unit so make it
file-local by adding the static keyword to its declaration/definition; update
the function signature for clusterGenNodesConfContent(...) to be static sds
clusterGenNodesConfContent(void) so the symbol is not exported while keeping the
implementation (calls to clusterGenNodesDescription, sdscatfmt and uses of
server.cluster->currentEpoch/lastVoteEpoch) unchanged.

47-47: Please get @core-team sign-off on this cluster persistence change set.

This rewires cluster config durability behavior in src/cluster_legacy.c, so it should get an architectural pass from the core team before merge.

As per coding guidelines, src/{cluster*.c,replication.c,rdb.c,aof.c}: Request @core-team architectural review for changes to cluster*.c, replication.c, rdb.c, or aof.c

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` at line 47, This change touches cluster durability
behavior in src/cluster_legacy.c and requires an explicit architectural sign-off
from the core team; update the PR and code to request that review by adding a
clear reviewer request in the PR description and add an in-file TODO comment
near the changed include or the related functions (e.g., any cluster
init/persistence functions in src/cluster_legacy.c) stating "REQ: `@core-team`
sign-off per coding guidelines for cluster/replication persistence changes", and
add the corresponding JIRA/issue tag or checklist entry so CI/reviewers cannot
merge without core-team approval.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/cluster_legacy.c`:
- Around line 6390-6399: The change made CLUSTER_TODO_FSYNC_CONFIG be handled
via clusterSaveConfigBackground (async) which breaks callers (e.g.,
clusterBumpConfigEpochWithoutConsensus, failover and slot-finalization code)
that relied on the flag meaning "persist before advertise"; restore the contract
by either (A) treating CLUSTER_TODO_FSYNC_CONFIG as synchronous: call
clusterSaveConfigOrDie(fsync) when flags include CLUSTER_TODO_FSYNC_CONFIG, or
(B) introduce a new flag (e.g., CLUSTER_TODO_FSYNC_BEFORE_ADVERTISE) and only
route that new flag to clusterSaveConfigOrDie while leaving legacy/best-effort
callers using clusterSaveConfigBackground; update callers
(clusterBumpConfigEpochWithoutConsensus and the failover/slot-finalization
paths) to use the synchronous semantic flag if they require crash-safe
persistence before broadcasting (and keep CLUSTER_TODO_BROADCAST_ALL handling
unchanged).

---

Outside diff comments:
In `@tests/unit/cluster/misc.tcl`:
- Around line 106-110: The test drains only node 0 (using the R 0 debug
bio-drain $BIO_CLUSTER_SAVE call) but then asserts logs/status on node -1 with
verify_log_message, which is racy; update the test to synchronize both nodes
before cross-node assertions by performing the equivalent drain/sync on node -1
as well (e.g., invoke the same debug bio-drain step or a cluster synchronization
for node -1) so that both R and verify_log_message checks operate on
deterministic, post-drain state for nodes 0 and -1.

---

Nitpick comments:
In `@src/bio.c`:
- Around line 334-345: The log rate limiter last_save_error_log in the
BIO_CLUSTER_SAVE branch prevents warnings after a recovery because it’s never
reset; modify the success branch in the BIO_CLUSTER_SAVE handling (the else
after clusterSaveConfigFromBio(...)) to clear or reset last_save_error_log (e.g.
set it to 0 or to server.unixtime - CONFIG_SAVE_LOG_ERROR_RATE) so a subsequent
failure will log again, and ensure
atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK,
memory_order_relaxed) remains unchanged; alternatively, gate the warning on the
previous value of server.cluster_config_bio_save_status to only suppress
repeated fails without masking recoveries.

In `@src/bio.h`:
- Line 46: bioCreateClusterConfigSaveJob currently hands sds data across threads
but the header prototype lacks ownership semantics; update the public
declaration for bioCreateClusterConfigSaveJob(sds content, bool do_fsync) in
src/bio.h with a clear doc comment that states whether the function takes
ownership of the sds (for example: "Takes ownership of 'content'; caller must
not free it after calling" or "Caller retains ownership; function makes an
internal copy"), describe expected life‑time/thread-transfer behavior and who is
responsible for freeing the sds, and mention any constraints on 'content'
(non-NULL, immutability expectations) and the effect of do_fsync; ensure callers
are audited to match the documented convention.

In `@src/cluster_legacy.c`:
- Around line 1039-1047: The function clusterGenNodesConfContent is used only
within this translation unit so make it file-local by adding the static keyword
to its declaration/definition; update the function signature for
clusterGenNodesConfContent(...) to be static sds
clusterGenNodesConfContent(void) so the symbol is not exported while keeping the
implementation (calls to clusterGenNodesDescription, sdscatfmt and uses of
server.cluster->currentEpoch/lastVoteEpoch) unchanged.
- Line 47: This change touches cluster durability behavior in
src/cluster_legacy.c and requires an explicit architectural sign-off from the
core team; update the PR and code to request that review by adding a clear
reviewer request in the PR description and add an in-file TODO comment near the
changed include or the related functions (e.g., any cluster init/persistence
functions in src/cluster_legacy.c) stating "REQ: `@core-team` sign-off per coding
guidelines for cluster/replication persistence changes", and add the
corresponding JIRA/issue tag or checklist entry so CI/reviewers cannot merge
without core-team approval.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: dc0dc17e-ca2e-4898-88d1-87c7d52f6ca9

📥 Commits

Reviewing files that changed from the base of the PR and between 5a604b3 and c3198d7.

📒 Files selected for processing (10)
  • src/bio.c
  • src/bio.h
  • src/cluster.h
  • src/cluster_legacy.c
  • src/debug.c
  • src/server.c
  • src/server.h
  • tests/support/cluster_util.tcl
  • tests/unit/cluster/misc.tcl
  • valkey.conf

Comment thread src/cluster_legacy.c

@ranshid ranshid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments.

I feel this is a very late point to include this in 9.1 release. Given the sensitivity of these changes and the potential impact on clustering, we might consider include this in the next release.
@enjoy-binbin what do you think?

Comment thread src/cluster_legacy.c Outdated
Comment thread src/cluster_legacy.c Outdated
Comment thread src/cluster_legacy.c
bool fsync = flags & CLUSTER_TODO_FSYNC_CONFIG;
if (server.cluster_configfile_save_behavior == CLUSTER_CONFIGFILE_SAVE_BEHAVIOR_SYNC) {
/* Sync mode: exit the process if saving fails. */
clusterSaveConfigOrDie(fsync);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we might have a pending async job in the queue, it might override the sync conf save here.
e.g:

  1. cluster-config-save-behavior=best-effort. Cluster event A → BIO job J1 queued (content snapshot S1).
  2. CONFIG SET cluster-config-save-behavior sync.
  3. Cluster event B → clusterBeforeSleep takes the sync branch → clusterSaveConfigOrDie writes content S2 (synchronously).
  4. BIO worker now processes J1 → overwrites nodes.conf with stale S1.
  5. nodes.conf is stale until the next cluster event triggers another save.

so suggest we drain the queue:

Suggested change
clusterSaveConfigOrDie(fsync);
bioDrainWorker(BIO_CLUSTER_SAVE);
clusterSaveConfigOrDie(fsync);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good cacth. Updated.

Comment thread src/bio.c
Comment thread tests/unit/cluster/misc.tcl
Comment thread tests/unit/cluster/misc.tcl
Comment thread src/cluster_legacy.c
Comment on lines +929 to +930
if (!from_bio) latencyAddSampleIfNeeded("cluster-config-open", latency);
if (!from_bio) latencyTraceIfNeeded(cluster, cluster_config_open, latency);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I agree this is not TS we should not prevent the user from being able to relaize he has disk related issues which impacts the config persistance lag. Maybe just a simple cluster_config_last_save_time (same pattern as rdb_last_save_time) which will help users identify when the config save is lagging too long?

enjoy-binbin and others added 2 commits May 18, 2026 19:21
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/cluster_legacy.c (1)

1131-1148: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Preserve errno across the cleanup path.

close()/unlink() can overwrite the original write/fsync/rename error here, so CLUSTER SAVECONFIG ends up formatting the wrong failure reason at Line 8080.

Suggested fix
 cleanup:
+    int saved_errno = errno;
     if (fd != -1) {
         latencyStartMonitor(latency);
         close(fd);
@@
     }
     sdsfree(tmpfilename);
     sdsfree(content);
+    if (retval == C_ERR) errno = saved_errno;
     return retval;
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` around lines 1131 - 1148, In the cleanup block around
fd/unlink (inside cluster_config save cleanup) preserve the original errno
before calling close() or unlink() so those syscalls don't clobber the real
error; capture errno into a local int (e.g. saved_errno) immediately before
calling close(fd) or unlink(tmpfilename), then after those calls restore errno =
saved_errno prior to returning retval so the original write/fsync/rename failure
is reported; update the section that handles fd != -1 and the retval == C_ERR
branch, ensuring saved_errno is set only once when the first failure occurs and
restored before sdsfree(tmpfilename)/sdsfree(content)/return.
♻️ Duplicate comments (3)
src/cluster_legacy.c (3)

6390-6399: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

CLUSTER_TODO_FSYNC_CONFIG still broadcasts before persistence in best-effort mode.

When cluster-config-save-behavior=best-effort, Line 6399 only queues the BIO write and CLUSTER_TODO_BROADCAST_ALL can still run immediately afterward. Callers such as clusterBumpConfigEpochWithoutConsensus() explicitly require “persist the configuration on disk before sending packets”, so this still regresses their contract. These paths need a synchronous save or a separate flag that preserves persist-before-advertise semantics.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` around lines 6390 - 6399, The best-effort branch
currently calls clusterSaveConfigBackground(fsync) which queues persistence but
still allows CLUSTER_TODO_BROADCAST_ALL to run immediately, breaking callers
that require on-disk persistence (e.g., clusterBumpConfigEpochWithoutConsensus).
Fix by honoring the persist-before-advertise contract: when flags include
CLUSTER_TODO_FSYNC_CONFIG (or introduce a dedicated flag like
CLUSTER_TODO_PERSIST_BEFORE_BROADCAST) call clusterSaveConfigOrDie(fsync)
(synchronous save + fsync) instead of clusterSaveConfigBackground, otherwise
keep the best-effort background save; update callers/flag checks accordingly to
reference CLUSTER_TODO_SAVE_CONFIG, CLUSTER_TODO_FSYNC_CONFIG,
clusterSaveConfigOrDie, clusterSaveConfigBackground, and
CLUSTER_TODO_BROADCAST_ALL.

1647-1650: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't ignore the final synchronous save failure during shutdown.

After the BIO drain, Line 1650 still lets shutdown continue on C_ERR. That can exit with a stale nodes.conf, which defeats the explicit persistence guarantee for shutdown.

Suggested fix
     serverLog(LL_NOTICE, "Saving the cluster configuration file before exiting.");
     bioDrainWorker(BIO_CLUSTER_SAVE);
-    clusterSaveConfig(false, true);
+    clusterSaveConfigOrDie(true);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` around lines 1647 - 1650, The code calls
bioDrainWorker(BIO_CLUSTER_SAVE) and then clusterSaveConfig(false, true) but
ignores a C_ERR return; change this so the return value of clusterSaveConfig is
checked and a failed save is handled (e.g., log an error with serverLog
including the failure context and prevent/abort normal shutdown or return a
non-zero exit) to guarantee nodes.conf is persisted; update the block around
bioDrainWorker/clusterSaveConfig to detect clusterSaveConfig(...) == C_ERR and
take a failure path instead of continuing shutdown.

7385-7406: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

CLUSTER INFO is still missing cluster_config_last_save_time.

This adds cluster_config_save_status, but not the last successful save timestamp promised by the PR contract. Without that field, an old "ok" is indistinguishable from a fresh one, so operators still can't tell whether BIO persistence is stalled.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cluster_legacy.c` around lines 7385 - 7406, Load the last-save timestamp
from server.cluster_config_last_save_time (using atomic_load_explicit with
memory_order_relaxed like config_bio_save_status), and add a new
"cluster_config_last_save_time:%U\r\n" entry to the sdscatfmt call that builds
info (using the same formatting convention as other epoch/timestamp fields, e.g.
casting to (unsigned long long)). Keep the existing cluster_config_save_status
logic but append the last-save time field so operators can see when the config
was last successfully persisted; update the sdscatfmt format string and its
argument list (in the same block where config_bio_save_status and info are
handled).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/cluster_legacy.c`:
- Around line 1131-1148: In the cleanup block around fd/unlink (inside
cluster_config save cleanup) preserve the original errno before calling close()
or unlink() so those syscalls don't clobber the real error; capture errno into a
local int (e.g. saved_errno) immediately before calling close(fd) or
unlink(tmpfilename), then after those calls restore errno = saved_errno prior to
returning retval so the original write/fsync/rename failure is reported; update
the section that handles fd != -1 and the retval == C_ERR branch, ensuring
saved_errno is set only once when the first failure occurs and restored before
sdsfree(tmpfilename)/sdsfree(content)/return.

---

Duplicate comments:
In `@src/cluster_legacy.c`:
- Around line 6390-6399: The best-effort branch currently calls
clusterSaveConfigBackground(fsync) which queues persistence but still allows
CLUSTER_TODO_BROADCAST_ALL to run immediately, breaking callers that require
on-disk persistence (e.g., clusterBumpConfigEpochWithoutConsensus). Fix by
honoring the persist-before-advertise contract: when flags include
CLUSTER_TODO_FSYNC_CONFIG (or introduce a dedicated flag like
CLUSTER_TODO_PERSIST_BEFORE_BROADCAST) call clusterSaveConfigOrDie(fsync)
(synchronous save + fsync) instead of clusterSaveConfigBackground, otherwise
keep the best-effort background save; update callers/flag checks accordingly to
reference CLUSTER_TODO_SAVE_CONFIG, CLUSTER_TODO_FSYNC_CONFIG,
clusterSaveConfigOrDie, clusterSaveConfigBackground, and
CLUSTER_TODO_BROADCAST_ALL.
- Around line 1647-1650: The code calls bioDrainWorker(BIO_CLUSTER_SAVE) and
then clusterSaveConfig(false, true) but ignores a C_ERR return; change this so
the return value of clusterSaveConfig is checked and a failed save is handled
(e.g., log an error with serverLog including the failure context and
prevent/abort normal shutdown or return a non-zero exit) to guarantee nodes.conf
is persisted; update the block around bioDrainWorker/clusterSaveConfig to detect
clusterSaveConfig(...) == C_ERR and take a failure path instead of continuing
shutdown.
- Around line 7385-7406: Load the last-save timestamp from
server.cluster_config_last_save_time (using atomic_load_explicit with
memory_order_relaxed like config_bio_save_status), and add a new
"cluster_config_last_save_time:%U\r\n" entry to the sdscatfmt call that builds
info (using the same formatting convention as other epoch/timestamp fields, e.g.
casting to (unsigned long long)). Keep the existing cluster_config_save_status
logic but append the last-save time field so operators can see when the config
was last successfully persisted; update the sdscatfmt format string and its
argument list (in the same block where config_bio_save_status and info are
handled).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5513efeb-cd56-4645-b208-43ad27100581

📥 Commits

Reviewing files that changed from the base of the PR and between c3198d7 and 23244a0.

📒 Files selected for processing (3)
  • src/bio.c
  • src/cluster_legacy.c
  • tests/unit/cluster/misc.tcl
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/bio.c
  • tests/unit/cluster/misc.tcl

@enjoy-binbin

Copy link
Copy Markdown
Member Author

@ranshid Thanks for the review. I agree that it is a late point, i don't mind postponing it.

@zuiderkwast zuiderkwast removed this from Valkey 9.1 Jul 3, 2026
@github-project-automation github-project-automation Bot moved this to Todo in Valkey 10 Jul 3, 2026
@zuiderkwast zuiderkwast moved this from Todo to Needs Review in Valkey 10 Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

major-decision-approved Major decision approved by TSC team run-extra-tests Run extra tests on this PR (Runs all tests from daily except valgrind and RESP)

Projects

Status: Needs Review

Development

Successfully merging this pull request may close these issues.

[BUG] Synchronous I/O in clusterSaveConfig

9 participants