Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode#2555
Use BIO thread for cluster config saving in cluster-config-save-behavior best-effort mode#2555enjoy-binbin wants to merge 20 commits into
Conversation
When the cluster changes, we need to persist the cluster configuration, if I/O is delayed or blocked, possibly by disk contention, this may result in large latencies on the main thread. We should avoid synchronous I/O from the main thread. So in this commit, we will try to bio to save the config file. We add a bio job and send a sds version of the config file, which does the synchronous save, so there is some eventually consistent version consistently stored on disk. This may break our previous assumption that nodes.conf is in sync and has the strong consistency. For shutdown and cluster saveconfig, we will wait for the bio job to get drained and trigger a new save in a sync way. Closes valkey-io#2424. Signed-off-by: Binbin <binloveplay1314@qq.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #2555 +/- ##
=========================================
Coverage 76.70% 76.70%
=========================================
Files 162 162
Lines 80674 80726 +52
=========================================
+ Hits 61880 61922 +42
- Misses 18794 18804 +10
🚀 New features to boost your workflow:
|
hpatro
left a comment
There was a problem hiding this comment.
High level thought:
We should make the clusterSaveConfig as async only otherwise we don't have any protection mechanism to disallow stale bio job to overwrite a freshly saved cluster config file from main thread.
| if (!from_bio) latencyAddSampleIfNeeded("cluster-config-open", latency); | ||
| if (!from_bio) latencyTraceIfNeeded(cluster, cluster_config_open, latency); |
There was a problem hiding this comment.
Is there any difference? Measuring latency should be the same for main thread/background thread I believe.
There was a problem hiding this comment.
latency is not currently thread-safe.
There was a problem hiding this comment.
While I agree this is not TS we should not prevent the user from being able to relaize he has disk related issues which impacts the config persistance lag. Maybe just a simple cluster_config_last_save_time (same pattern as rdb_last_save_time) which will help users identify when the config save is lagging too long?
There was a problem hiding this comment.
Good point, added a new cluster_config_last_save_time cluster info field.
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com> Signed-off-by: Binbin <binloveplay1314@qq.com>
| bioSubmitJob(BIO_RDB_SAVE, job); | ||
| } | ||
|
|
||
| void bioCreateClusterConfigSaveJob(sds content, int do_fsync) { |
There was a problem hiding this comment.
Who owns/free’s content? bioCreateClusterConfigSaveJob(sds content, int do_fsync) enqueues an sds into job->cluster_save_args.content, then the BIO worker calls clusterSaveConfigFromBio(...). is there a clear free path for content afterwards—Would you please document ownership and ensure a single, deterministic free site to avoid leaks or double-frees.
There was a problem hiding this comment.
sorry for the dealy, clusterSaveConfigImpl will own and free the content.
| latencyStartMonitor(latency); | ||
| while (offset < content_size) { | ||
| written_bytes = write(fd, ci + offset, content_size - offset); | ||
| written_bytes = write(fd, content + offset, content_size - offset); |
There was a problem hiding this comment.
If multiple saves are enqueued, can older content overwrite newer ones and amplify I/O. Can we Coalesce jobs so only the latest content is written (drop older items). Do we need some sort of versioning/epoch so the older jobs are dropped.
There was a problem hiding this comment.
good point, i guess we can do it later?
| } else if (job_type == BIO_RDB_SAVE) { | ||
| replicaReceiveRDBFromPrimaryToDisk(job->save_to_disk_args.conn, job->save_to_disk_args.is_dual_channel); | ||
| } else if (job_type == BIO_CLUSTER_SAVE) { | ||
| if (clusterSaveConfigFromBio(job->cluster_save_args.content, job->cluster_save_args.do_fsync) == C_ERR) { |
There was a problem hiding this comment.
Do we need to introduce exponential backoff and retry to mitigate transient errors. Can we include observability and metrics around the type of error we notice while trying to save the file/directory
There was a problem hiding this comment.
so you want something like aof_last_write_status in the INFO fields?
There was a problem hiding this comment.
Do we need to introduce exponential backoff and retry to mitigate transient errors.
Cluster configuration file updates should occur infrequently, so i think we are ok?
Can we include observability and metrics around the type of error we notice while trying to save the file/directory
Added a new cluster_config_save_status filed in CLUSTER INFO, does this slove your concern?
Sorry for the delay reply, these related PRs have indeed been dragging on for quite a while, let's pick them back up and merge them into 9.1.
|
Core team meeting:
|
|
Can we target this fix for upcoming patch release?. |
|
one concern about shutdown, I think we should call |
Signed-off-by: Binbin <binloveplay1314@qq.com>
|
@cherukum-Amazon sorry for the dealy, i somehow lost the context a while ago, i will try to refresh it this week. Let's start working on #1032 first and try to push it forward.
yes, we do call bioDrainWorker in finishShutdown. |
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
hpatro
left a comment
There was a problem hiding this comment.
Let's make a decision on this @valkey-io/core-team. Could you folks vote if we can support cluster config persistence in bio 👍 / 👎 ?
Risks with moving the cluster node configuration persistence to bio (reiterating some of the discussions we had):
- lastVoteEpoch is no longer persisted synchronously, potential double voting issue can surface on crash. This issue is similar to not exiting the process on failure to persist the config file.
- Put this change behind
cluster-persist-config->best-effort(name to be finalized) config getting introduced in #3372.
I believe operators are fine with offloading this to BIO to avoid dependency on the main thread which can get stuck and the voting correctness issue is of lesser importance.
|
I thought this was already covered by the major decision we did for #3372. Didn't we discuss these two PRs togeher? |
Seems like a miss we didn't mark this PR as major decision approved. Let me update it. |
|
@hpatro I saw you just updated this pr to major-decision-approved, do we need to discuss it in next Monday meeting? (I found it is in the next Monday meeting agenda) |
Apparently this was discussed alongside #3372 and was approved over the Monday contributor's meeting. We were just missing the right label. We will wait for binbin to wrap up the changes. |
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
📝 WalkthroughWalkthroughMoves cluster config saves to a BIO background worker. Adds BIO_CLUSTER_SAVE, bioCreateClusterConfigSaveJob(), clusterSaveConfigFromBio(), refactors save logic for BIO/main-thread contexts, tracks atomic save status exposed in CLUSTER INFO, adds DEBUG BIO-DRAIN, and updates tests and docs. ChangesCluster Config BIO Save
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/unit/cluster/misc.tcl (1)
106-110:⚠️ Potential issue | 🟠 Major | ⚡ Quick winSynchronize both nodes before asserting cross-node logs/status.
Line 106 and Line 121 drain only node
0, but subsequent assertions read node-1state too. In best-effort mode this is racy and can make the test flaky.Suggested fix
- R 0 debug bio-drain $BIO_CLUSTER_SAVE + R 0 debug bio-drain $BIO_CLUSTER_SAVE + R 1 debug bio-drain $BIO_CLUSTER_SAVE @@ - R 0 debug bio-drain $BIO_CLUSTER_SAVE + R 0 debug bio-drain $BIO_CLUSTER_SAVE + R 1 debug bio-drain $BIO_CLUSTER_SAVEAs per coding guidelines "Avoid timing-dependent tests; use proper synchronization (test reliability)".
Also applies to: 121-130
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/cluster/misc.tcl` around lines 106 - 110, The test drains only node 0 (using the R 0 debug bio-drain $BIO_CLUSTER_SAVE call) but then asserts logs/status on node -1 with verify_log_message, which is racy; update the test to synchronize both nodes before cross-node assertions by performing the equivalent drain/sync on node -1 as well (e.g., invoke the same debug bio-drain step or a cluster synchronization for node -1) so that both R and verify_log_message checks operate on deterministic, post-drain state for nodes 0 and -1.
🧹 Nitpick comments (4)
src/bio.h (1)
46-46: ⚡ Quick winClarify
contentownership in this public BIO API.
bioCreateClusterConfigSaveJob(sds content, bool do_fsync)hands data across threads; please document whether ownership ofcontentis transferred or retained to prevent leaks/UAF at call sites.As per coding guidelines, "Document why code exists, not just what it does; document all functions in C code".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/bio.h` at line 46, bioCreateClusterConfigSaveJob currently hands sds data across threads but the header prototype lacks ownership semantics; update the public declaration for bioCreateClusterConfigSaveJob(sds content, bool do_fsync) in src/bio.h with a clear doc comment that states whether the function takes ownership of the sds (for example: "Takes ownership of 'content'; caller must not free it after calling" or "Caller retains ownership; function makes an internal copy"), describe expected life‑time/thread-transfer behavior and who is responsible for freeing the sds, and mention any constraints on 'content' (non-NULL, immutability expectations) and the effect of do_fsync; ensure callers are audited to match the documented convention.src/bio.c (1)
334-345: ⚡ Quick winReset the log suppressor after a successful save.
last_save_error_lognever gets cleared on success, so a fail → recover → fail sequence inside 30 seconds stays silent on the second failure. That hides the transition operators usually care about most. Reset the limiter on success, or gate the warning on the previouscluster_config_bio_save_statusas well.Suggested tweak
} else if (job_type == BIO_CLUSTER_SAVE) { static time_t last_save_error_log = 0; if (clusterSaveConfigFromBio(job->cluster_save_args.content, job->cluster_save_args.do_fsync) == C_ERR) { /* Limit logging rate to 1 line per CONFIG_SAVE_LOG_ERROR_RATE seconds. */ if ((server.unixtime - last_save_error_log) > CONFIG_SAVE_LOG_ERROR_RATE) { serverLog(LL_WARNING, "Failed to save the cluster config file in background. Cluster config " "updated even though writing the cluster config file to disk failed."); last_save_error_log = server.unixtime; } atomic_store_explicit(&server.cluster_config_bio_save_status, C_ERR, memory_order_relaxed); } else { atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK, memory_order_relaxed); + last_save_error_log = 0; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/bio.c` around lines 334 - 345, The log rate limiter last_save_error_log in the BIO_CLUSTER_SAVE branch prevents warnings after a recovery because it’s never reset; modify the success branch in the BIO_CLUSTER_SAVE handling (the else after clusterSaveConfigFromBio(...)) to clear or reset last_save_error_log (e.g. set it to 0 or to server.unixtime - CONFIG_SAVE_LOG_ERROR_RATE) so a subsequent failure will log again, and ensure atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK, memory_order_relaxed) remains unchanged; alternatively, gate the warning on the previous value of server.cluster_config_bio_save_status to only suppress repeated fails without masking recoveries.src/cluster_legacy.c (2)
1039-1047: ⚡ Quick winMake this helper file-local.
clusterGenNodesConfContent()is only used insidesrc/cluster_legacy.c, so exporting the symbol unnecessarily widens the surface area.As per coding guidelines, `src/**/*.{c,h}`: `Use static keyword for file-local functions in C code`♻️ Proposed fix
-sds clusterGenNodesConfContent(void) { +static sds clusterGenNodesConfContent(void) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` around lines 1039 - 1047, The function clusterGenNodesConfContent is used only within this translation unit so make it file-local by adding the static keyword to its declaration/definition; update the function signature for clusterGenNodesConfContent(...) to be static sds clusterGenNodesConfContent(void) so the symbol is not exported while keeping the implementation (calls to clusterGenNodesDescription, sdscatfmt and uses of server.cluster->currentEpoch/lastVoteEpoch) unchanged.
47-47: Please get@core-teamsign-off on this cluster persistence change set.This rewires cluster config durability behavior in
src/cluster_legacy.c, so it should get an architectural pass from the core team before merge.As per coding guidelines,
src/{cluster*.c,replication.c,rdb.c,aof.c}:Request@core-teamarchitectural review for changes to cluster*.c, replication.c, rdb.c, or aof.c🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` at line 47, This change touches cluster durability behavior in src/cluster_legacy.c and requires an explicit architectural sign-off from the core team; update the PR and code to request that review by adding a clear reviewer request in the PR description and add an in-file TODO comment near the changed include or the related functions (e.g., any cluster init/persistence functions in src/cluster_legacy.c) stating "REQ: `@core-team` sign-off per coding guidelines for cluster/replication persistence changes", and add the corresponding JIRA/issue tag or checklist entry so CI/reviewers cannot merge without core-team approval.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cluster_legacy.c`:
- Around line 6390-6399: The change made CLUSTER_TODO_FSYNC_CONFIG be handled
via clusterSaveConfigBackground (async) which breaks callers (e.g.,
clusterBumpConfigEpochWithoutConsensus, failover and slot-finalization code)
that relied on the flag meaning "persist before advertise"; restore the contract
by either (A) treating CLUSTER_TODO_FSYNC_CONFIG as synchronous: call
clusterSaveConfigOrDie(fsync) when flags include CLUSTER_TODO_FSYNC_CONFIG, or
(B) introduce a new flag (e.g., CLUSTER_TODO_FSYNC_BEFORE_ADVERTISE) and only
route that new flag to clusterSaveConfigOrDie while leaving legacy/best-effort
callers using clusterSaveConfigBackground; update callers
(clusterBumpConfigEpochWithoutConsensus and the failover/slot-finalization
paths) to use the synchronous semantic flag if they require crash-safe
persistence before broadcasting (and keep CLUSTER_TODO_BROADCAST_ALL handling
unchanged).
---
Outside diff comments:
In `@tests/unit/cluster/misc.tcl`:
- Around line 106-110: The test drains only node 0 (using the R 0 debug
bio-drain $BIO_CLUSTER_SAVE call) but then asserts logs/status on node -1 with
verify_log_message, which is racy; update the test to synchronize both nodes
before cross-node assertions by performing the equivalent drain/sync on node -1
as well (e.g., invoke the same debug bio-drain step or a cluster synchronization
for node -1) so that both R and verify_log_message checks operate on
deterministic, post-drain state for nodes 0 and -1.
---
Nitpick comments:
In `@src/bio.c`:
- Around line 334-345: The log rate limiter last_save_error_log in the
BIO_CLUSTER_SAVE branch prevents warnings after a recovery because it’s never
reset; modify the success branch in the BIO_CLUSTER_SAVE handling (the else
after clusterSaveConfigFromBio(...)) to clear or reset last_save_error_log (e.g.
set it to 0 or to server.unixtime - CONFIG_SAVE_LOG_ERROR_RATE) so a subsequent
failure will log again, and ensure
atomic_store_explicit(&server.cluster_config_bio_save_status, C_OK,
memory_order_relaxed) remains unchanged; alternatively, gate the warning on the
previous value of server.cluster_config_bio_save_status to only suppress
repeated fails without masking recoveries.
In `@src/bio.h`:
- Line 46: bioCreateClusterConfigSaveJob currently hands sds data across threads
but the header prototype lacks ownership semantics; update the public
declaration for bioCreateClusterConfigSaveJob(sds content, bool do_fsync) in
src/bio.h with a clear doc comment that states whether the function takes
ownership of the sds (for example: "Takes ownership of 'content'; caller must
not free it after calling" or "Caller retains ownership; function makes an
internal copy"), describe expected life‑time/thread-transfer behavior and who is
responsible for freeing the sds, and mention any constraints on 'content'
(non-NULL, immutability expectations) and the effect of do_fsync; ensure callers
are audited to match the documented convention.
In `@src/cluster_legacy.c`:
- Around line 1039-1047: The function clusterGenNodesConfContent is used only
within this translation unit so make it file-local by adding the static keyword
to its declaration/definition; update the function signature for
clusterGenNodesConfContent(...) to be static sds
clusterGenNodesConfContent(void) so the symbol is not exported while keeping the
implementation (calls to clusterGenNodesDescription, sdscatfmt and uses of
server.cluster->currentEpoch/lastVoteEpoch) unchanged.
- Line 47: This change touches cluster durability behavior in
src/cluster_legacy.c and requires an explicit architectural sign-off from the
core team; update the PR and code to request that review by adding a clear
reviewer request in the PR description and add an in-file TODO comment near the
changed include or the related functions (e.g., any cluster init/persistence
functions in src/cluster_legacy.c) stating "REQ: `@core-team` sign-off per coding
guidelines for cluster/replication persistence changes", and add the
corresponding JIRA/issue tag or checklist entry so CI/reviewers cannot merge
without core-team approval.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: dc0dc17e-ca2e-4898-88d1-87c7d52f6ca9
📒 Files selected for processing (10)
src/bio.csrc/bio.hsrc/cluster.hsrc/cluster_legacy.csrc/debug.csrc/server.csrc/server.htests/support/cluster_util.tcltests/unit/cluster/misc.tclvalkey.conf
ranshid
left a comment
There was a problem hiding this comment.
Some comments.
I feel this is a very late point to include this in 9.1 release. Given the sensitivity of these changes and the potential impact on clustering, we might consider include this in the next release.
@enjoy-binbin what do you think?
| bool fsync = flags & CLUSTER_TODO_FSYNC_CONFIG; | ||
| if (server.cluster_configfile_save_behavior == CLUSTER_CONFIGFILE_SAVE_BEHAVIOR_SYNC) { | ||
| /* Sync mode: exit the process if saving fails. */ | ||
| clusterSaveConfigOrDie(fsync); |
There was a problem hiding this comment.
Since we might have a pending async job in the queue, it might override the sync conf save here.
e.g:
- cluster-config-save-behavior=best-effort. Cluster event A → BIO job J1 queued (content snapshot S1).
- CONFIG SET cluster-config-save-behavior sync.
- Cluster event B → clusterBeforeSleep takes the sync branch → clusterSaveConfigOrDie writes content S2 (synchronously).
- BIO worker now processes J1 → overwrites nodes.conf with stale S1.
- nodes.conf is stale until the next cluster event triggers another save.
so suggest we drain the queue:
| clusterSaveConfigOrDie(fsync); | |
| bioDrainWorker(BIO_CLUSTER_SAVE); | |
| clusterSaveConfigOrDie(fsync); |
There was a problem hiding this comment.
Good cacth. Updated.
| if (!from_bio) latencyAddSampleIfNeeded("cluster-config-open", latency); | ||
| if (!from_bio) latencyTraceIfNeeded(cluster, cluster_config_open, latency); |
There was a problem hiding this comment.
While I agree this is not TS we should not prevent the user from being able to relaize he has disk related issues which impacts the config persistance lag. Maybe just a simple cluster_config_last_save_time (same pattern as rdb_last_save_time) which will help users identify when the config save is lagging too long?
Co-authored-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/cluster_legacy.c (1)
1131-1148:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winPreserve
errnoacross the cleanup path.
close()/unlink()can overwrite the original write/fsync/rename error here, soCLUSTER SAVECONFIGends up formatting the wrong failure reason at Line 8080.Suggested fix
cleanup: + int saved_errno = errno; if (fd != -1) { latencyStartMonitor(latency); close(fd); @@ } sdsfree(tmpfilename); sdsfree(content); + if (retval == C_ERR) errno = saved_errno; return retval; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` around lines 1131 - 1148, In the cleanup block around fd/unlink (inside cluster_config save cleanup) preserve the original errno before calling close() or unlink() so those syscalls don't clobber the real error; capture errno into a local int (e.g. saved_errno) immediately before calling close(fd) or unlink(tmpfilename), then after those calls restore errno = saved_errno prior to returning retval so the original write/fsync/rename failure is reported; update the section that handles fd != -1 and the retval == C_ERR branch, ensuring saved_errno is set only once when the first failure occurs and restored before sdsfree(tmpfilename)/sdsfree(content)/return.
♻️ Duplicate comments (3)
src/cluster_legacy.c (3)
6390-6399:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift
CLUSTER_TODO_FSYNC_CONFIGstill broadcasts before persistence in best-effort mode.When
cluster-config-save-behavior=best-effort, Line 6399 only queues the BIO write andCLUSTER_TODO_BROADCAST_ALLcan still run immediately afterward. Callers such asclusterBumpConfigEpochWithoutConsensus()explicitly require “persist the configuration on disk before sending packets”, so this still regresses their contract. These paths need a synchronous save or a separate flag that preserves persist-before-advertise semantics.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` around lines 6390 - 6399, The best-effort branch currently calls clusterSaveConfigBackground(fsync) which queues persistence but still allows CLUSTER_TODO_BROADCAST_ALL to run immediately, breaking callers that require on-disk persistence (e.g., clusterBumpConfigEpochWithoutConsensus). Fix by honoring the persist-before-advertise contract: when flags include CLUSTER_TODO_FSYNC_CONFIG (or introduce a dedicated flag like CLUSTER_TODO_PERSIST_BEFORE_BROADCAST) call clusterSaveConfigOrDie(fsync) (synchronous save + fsync) instead of clusterSaveConfigBackground, otherwise keep the best-effort background save; update callers/flag checks accordingly to reference CLUSTER_TODO_SAVE_CONFIG, CLUSTER_TODO_FSYNC_CONFIG, clusterSaveConfigOrDie, clusterSaveConfigBackground, and CLUSTER_TODO_BROADCAST_ALL.
1647-1650:⚠️ Potential issue | 🟠 Major | ⚡ Quick winDon't ignore the final synchronous save failure during shutdown.
After the BIO drain, Line 1650 still lets shutdown continue on
C_ERR. That can exit with a stalenodes.conf, which defeats the explicit persistence guarantee for shutdown.Suggested fix
serverLog(LL_NOTICE, "Saving the cluster configuration file before exiting."); bioDrainWorker(BIO_CLUSTER_SAVE); - clusterSaveConfig(false, true); + clusterSaveConfigOrDie(true);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` around lines 1647 - 1650, The code calls bioDrainWorker(BIO_CLUSTER_SAVE) and then clusterSaveConfig(false, true) but ignores a C_ERR return; change this so the return value of clusterSaveConfig is checked and a failed save is handled (e.g., log an error with serverLog including the failure context and prevent/abort normal shutdown or return a non-zero exit) to guarantee nodes.conf is persisted; update the block around bioDrainWorker/clusterSaveConfig to detect clusterSaveConfig(...) == C_ERR and take a failure path instead of continuing shutdown.
7385-7406:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift
CLUSTER INFOis still missingcluster_config_last_save_time.This adds
cluster_config_save_status, but not the last successful save timestamp promised by the PR contract. Without that field, an old"ok"is indistinguishable from a fresh one, so operators still can't tell whether BIO persistence is stalled.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cluster_legacy.c` around lines 7385 - 7406, Load the last-save timestamp from server.cluster_config_last_save_time (using atomic_load_explicit with memory_order_relaxed like config_bio_save_status), and add a new "cluster_config_last_save_time:%U\r\n" entry to the sdscatfmt call that builds info (using the same formatting convention as other epoch/timestamp fields, e.g. casting to (unsigned long long)). Keep the existing cluster_config_save_status logic but append the last-save time field so operators can see when the config was last successfully persisted; update the sdscatfmt format string and its argument list (in the same block where config_bio_save_status and info are handled).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/cluster_legacy.c`:
- Around line 1131-1148: In the cleanup block around fd/unlink (inside
cluster_config save cleanup) preserve the original errno before calling close()
or unlink() so those syscalls don't clobber the real error; capture errno into a
local int (e.g. saved_errno) immediately before calling close(fd) or
unlink(tmpfilename), then after those calls restore errno = saved_errno prior to
returning retval so the original write/fsync/rename failure is reported; update
the section that handles fd != -1 and the retval == C_ERR branch, ensuring
saved_errno is set only once when the first failure occurs and restored before
sdsfree(tmpfilename)/sdsfree(content)/return.
---
Duplicate comments:
In `@src/cluster_legacy.c`:
- Around line 6390-6399: The best-effort branch currently calls
clusterSaveConfigBackground(fsync) which queues persistence but still allows
CLUSTER_TODO_BROADCAST_ALL to run immediately, breaking callers that require
on-disk persistence (e.g., clusterBumpConfigEpochWithoutConsensus). Fix by
honoring the persist-before-advertise contract: when flags include
CLUSTER_TODO_FSYNC_CONFIG (or introduce a dedicated flag like
CLUSTER_TODO_PERSIST_BEFORE_BROADCAST) call clusterSaveConfigOrDie(fsync)
(synchronous save + fsync) instead of clusterSaveConfigBackground, otherwise
keep the best-effort background save; update callers/flag checks accordingly to
reference CLUSTER_TODO_SAVE_CONFIG, CLUSTER_TODO_FSYNC_CONFIG,
clusterSaveConfigOrDie, clusterSaveConfigBackground, and
CLUSTER_TODO_BROADCAST_ALL.
- Around line 1647-1650: The code calls bioDrainWorker(BIO_CLUSTER_SAVE) and
then clusterSaveConfig(false, true) but ignores a C_ERR return; change this so
the return value of clusterSaveConfig is checked and a failed save is handled
(e.g., log an error with serverLog including the failure context and
prevent/abort normal shutdown or return a non-zero exit) to guarantee nodes.conf
is persisted; update the block around bioDrainWorker/clusterSaveConfig to detect
clusterSaveConfig(...) == C_ERR and take a failure path instead of continuing
shutdown.
- Around line 7385-7406: Load the last-save timestamp from
server.cluster_config_last_save_time (using atomic_load_explicit with
memory_order_relaxed like config_bio_save_status), and add a new
"cluster_config_last_save_time:%U\r\n" entry to the sdscatfmt call that builds
info (using the same formatting convention as other epoch/timestamp fields, e.g.
casting to (unsigned long long)). Keep the existing cluster_config_save_status
logic but append the last-save time field so operators can see when the config
was last successfully persisted; update the sdscatfmt format string and its
argument list (in the same block where config_bio_save_status and info are
handled).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5513efeb-cd56-4645-b208-43ad27100581
📒 Files selected for processing (3)
src/bio.csrc/cluster_legacy.ctests/unit/cluster/misc.tcl
🚧 Files skipped from review as they are similar to previous changes (2)
- src/bio.c
- tests/unit/cluster/misc.tcl
|
@ranshid Thanks for the review. I agree that it is a late point, i don't mind postponing it. |
When the cluster changes, we need to persist the cluster configuration.
Currently, nodes.conf is saved synchronously in the main thread during
clusterBeforeSleep, if I/O is delayed or blocked, possibly by disk
contention, this may result in large latencies on the main thread that
affect client requests.
We should avoid synchronous I/O from the main thread.
In this commit, we will try to use bio to save the config file when
cluster-config-save-behavior is set to best-effort. We add a new bio job
and send a sds version of the config file, which does the synchronous save,
so there is some eventually consistent version consistently stored on disk.
For shutdown and cluster saveconfig, we will wait for the bio job to get
drained and trigger a new save in a sync way.
New cluster info fields:
New DEBUG BIO-DRAIN subcommand to drain the bio jobs.
Closes #2424.