kvserver: deflake TestReadLoadMetricAccounting#159877
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Dec 19, 2025
Merged
kvserver: deflake TestReadLoadMetricAccounting#159877craig[bot] merged 1 commit intocockroachdb:masterfrom
craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
Member
|
This change is |
6a87d6a to
eee13a8
Compare
Wrap each test case in testutils.SucceedsSoon to handle interference from background activity (e.g., async stats recording from lease upgrades). If stats don't match expectations due to background writes, the test resets and retries instead of failing immediately. Fixes: cockroachdb#159719 Release note: None
eee13a8 to
7be2f39
Compare
pav-kv
approved these changes
Dec 19, 2025
Member
Author
|
bors r+ Not sure what's wrong with the pull_request build - looks like an infra problem. Asked here |
Contributor
Member
Author
|
blathers backport 26.1 25.4 |
|
Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches. Issue #159719: branch-release-25.4, branch-release-26.1. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
This was referenced Feb 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestReadLoadMetricAccountinghas a history of flaking due to lease-related writes interfering with load metric measurements.Issue #141716 (and #141586) identified the same failure signature:
The root cause was identified by @pav-kv: an "unexpected" leader lease upgrade write was interfering with the test's write bytes measurements. PR #141843 added
tc.MaybeWaitForLeaseUpgrade()to wait for lease upgrades before starting measurements.The fix from #141843 IS present in the failing SHA. However, the test still flaked with the same error signature (85 write bytes when expecting 0).
The logs show:
The race condition is subtle:
MaybeWaitForLeaseUpgrade()waits untilFindRangeLeaseEx()reports the lease is upgraded, but it does not guarantee that the write bytes have been recorded to load stats. This is because stats are recorded "awkwardly late" on the client goroutine (SendWithWriteBytes).The fix:
SucceedsSoonRelated Issues/PRs
This is more robust than trying to synchronize with specific background operations because it handles any source of interference, not just lease upgrades.
Epic: none
Closes #159719.