Skip to content

db: add RangeKeyIteratorStats#1871

Merged
jbowens merged 1 commit intocockroachdb:masterfrom
jbowens:iter-rkstats
Dec 12, 2022
Merged

db: add RangeKeyIteratorStats#1871
jbowens merged 1 commit intocockroachdb:masterfrom
jbowens:iter-rkstats

Conversation

@jbowens
Copy link
Copy Markdown
Contributor

@jbowens jbowens commented Aug 11, 2022

Expand the IteratorStats structure to include a few statistics about range
key iteration.

Close #1848.

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@jbowens jbowens force-pushed the iter-rkstats branch 2 times, most recently from 002fac9 to 83c5e49 Compare August 11, 2022 20:00
Copy link
Copy Markdown
Contributor Author

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little at a loss at these CI failures but will keep digging.

Edit: Turned out to be nondeterminism from -tags invariants.

Reviewable status: 0 of 3 files reviewed, all discussions resolved

@jbowens jbowens force-pushed the iter-rkstats branch 3 times, most recently from dadaadc to bd7ba6b Compare August 23, 2022 18:06
@jbowens jbowens force-pushed the iter-rkstats branch 4 times, most recently from a1b9827 to edd543d Compare November 10, 2022 17:25
@jbowens jbowens force-pushed the iter-rkstats branch 5 times, most recently from 79d2a09 to f824072 Compare December 9, 2022 19:30
@jbowens jbowens requested review from a team and sumeerbhola December 9, 2022 19:39
@sumeerbhola
Copy link
Copy Markdown
Collaborator

Has this really been sitting unreviewed since August?

Copy link
Copy Markdown
Collaborator

@sumeerbhola sumeerbhola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

What was the underlying cause of the non-determinism?

Reviewed 5 of 5 files at r1.
Reviewable status: all files reviewed (commit messages unreviewed), 2 unresolved discussions (waiting on @jbowens)


iterator.go line 139 at r1 (raw file):

	Count int
	// ContainedPoints records the number of point keys encountered within the
	// bounds of a range key.

this is just about the bounds and nothing about masking, so the point keys could be more recent?


iterator.go line 141 at r1 (raw file):

	// bounds of a range key.
	ContainedPoints int
	// SkippedPoints records the count of the subset of CoveredPoints point keys

... subset of ContainedPoints ...?

Copy link
Copy Markdown
Contributor Author

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this really been sitting unreviewed since August?

Ha, no, it's been sitting with CI test failures since August. I had a hard time debugging the CI test failures and then got caught up in the 22.2 rush.

What was the underlying cause of the non-determinism?

it was the disabling of the noop try-seek-using-next optimization, which uses the iterator pointer. a noop would result in fewer encountered point keys

TFTR!

Reviewable status: all files reviewed (commit messages unreviewed), 2 unresolved discussions (waiting on @sumeerbhola)


iterator.go line 139 at r1 (raw file):

Previously, sumeerbhola wrote…

this is just about the bounds and nothing about masking, so the point keys could be more recent?

yeah, added another sentence to clarify that


iterator.go line 141 at r1 (raw file):

Previously, sumeerbhola wrote…

... subset of ContainedPoints ...?

Done.

Expand the IteratorStats structure to include a few statistics about range
key iteration.

Informs cockroachdb#1848.
@jbowens jbowens merged commit 6311f7a into cockroachdb:master Dec 12, 2022
@jbowens jbowens deleted the iter-rkstats branch December 12, 2022 17:19
jbowens added a commit to jbowens/cockroach that referenced this pull request Dec 27, 2022
Include the new, low-level Pebble range key iterator stats (introduced in
cockroachdb/pebble#1871) in roachpb.ScanStats.

Informs cockroachdb#77580.
Close cockroachdb#93326.

Epic: None
Release note: None
craig bot pushed a commit to cockroachdb/cockroach that referenced this pull request Jan 3, 2023
94010: sql/importer: unskip TestCSVImportCanBeResumed r=rafiss a=stevendanna

This unskips the test and adds a memory limit. The test completed 1500+ runs under stress. I believe #93782 resolved the main source of flakiness here.

Fixes #91828

Release note: None

94345: storage: include range key stats in ScanStats r=erikgrinaker a=jbowens

Include the new, low-level Pebble range key iterator stats (introduced in cockroachdb/pebble#1871) in roachpb.ScanStats.

Informs #77580.
Close #93326.

Epic: None
Release note: None

94458: logictest: fix flakes from mixed version testing r=ZhouXing19 a=rafiss

fixes #92637

The main fix was to wait for each node to be reachable before beginning the upgrade process. This also includes a version bump that has better logging to make it easier to debug.

Release note: None

Co-authored-by: Steven Danna <danna@cockroachlabs.com>
Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
blathers-crl bot pushed a commit to cockroachdb/cockroach that referenced this pull request Jan 3, 2023
Include the new, low-level Pebble range key iterator stats (introduced in
cockroachdb/pebble#1871) in roachpb.ScanStats.

Informs #77580.
Close #93326.

Epic: None
Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

db: improve range key observability

3 participants