Skip to content

kvfollowerreadsccl: maybe deflake TestBoundedStalenessDataDriven#157915

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
stevendanna:ssd/push-mrzzklzwkpmw
Nov 17, 2025
Merged

kvfollowerreadsccl: maybe deflake TestBoundedStalenessDataDriven#157915
craig[bot] merged 1 commit intocockroachdb:masterfrom
stevendanna:ssd/push-mrzzklzwkpmw

Conversation

@stevendanna
Copy link
Copy Markdown
Collaborator

This test determines what events occur by parsing the trace. In some cases, the parsing it was using to determine a "local read followed by remote leaseholder read" didn't account for changes in the potential trace messages encountered when leader leases are enabled.

Here, I widen the scope of the trace parsing. Locally under stress this elliminated the previously encountered failure:

    datadriven.go:357: ... SNIP ... boundedstaleness/single_row:24: still running after 10.000889738s
    ... SNIP ...
    boundedstaleness_test.go:405: condition failed to evaluate within 45s: from boundedstaleness_test.go:436: not yet a match, output:
        1
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read
    datadriven.go:343:

Fixes #154710
Release note: None

This test determines what events occur by parsing the trace. In some cases, the
parsing it was using to determine a "local read followed by remote leaseholder
read" didn't account for changes in the potential trace messages encountered
when leader leases are enabled.

Here, I widen the scope of the trace parsing. Locally under stress this
elliminated the previously encountered failure:

```
    datadriven.go:357: ... SNIP ... boundedstaleness/single_row:24: still running after 10.000889738s
    ... SNIP ...
    boundedstaleness_test.go:405: condition failed to evaluate within 45s: from boundedstaleness_test.go:436: not yet a match, output:
        1
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read
    datadriven.go:343:
```

Fixes cockroachdb#154710
Release note: None
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Collaborator

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@arulajmani reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @miraradeva)

Copy link
Copy Markdown
Contributor

@miraradeva miraradeva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@miraradeva reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: :shipit: complete! 2 of 0 LGTMs obtained (waiting on @stevendanna)


pkg/ccl/kvccl/kvfollowerreadsccl/boundedstaleness_test.go line 50 at r1 (raw file):

// fullTraceDebug is a flag that controls whether full traces are printed in the
// case of some errors.
const fullTraceDebug = false

This is just for debugging the test, right?

@stevendanna
Copy link
Copy Markdown
Collaborator Author

TFTR!

I'm going to merge this, but I think we probably want to follow up more here. Relying on the trace is pretty brittle and currently we are actually failing to catch cases that violate some of the tests assumptions. For example, occasionally

SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('1μs', true) WHERE pk = 1

Goes to the leaseholder because of an uninitialised replica. We don't catch that right now only because of the loosy-goosy way that we parse the trace.

@stevendanna
Copy link
Copy Markdown
Collaborator Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Nov 17, 2025

@craig craig bot merged commit 0b875de into cockroachdb:master Nov 17, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ccl/kvccl/kvfollowerreadsccl: TestBoundedStalenessDataDriven failed

4 participants