Skip to content

storage: expose additional engine metrics#46558

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
jbowens:engine-metrics
Mar 26, 2020
Merged

storage: expose additional engine metrics#46558
craig[bot] merged 1 commit intocockroachdb:masterfrom
jbowens:engine-metrics

Conversation

@jbowens
Copy link
Copy Markdown
Contributor

@jbowens jbowens commented Mar 25, 2020

Release justification: low-risk update to new functionality

Fixes #43965

Release note: none

@jbowens jbowens added the first-pr Use to mark the first PR sent by a contributor / team member. Reviewers should be mindful of this. label Mar 25, 2020
@jbowens jbowens requested a review from itsbilal March 25, 2020 14:40
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Collaborator

@petermattis petermattis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very much like what I was expecting. I have a few small comments below.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @itsbilal and @jbowens)


pkg/kv/kvserver/metrics.go, line 354 at r1 (raw file):

	}
	metaRdbCompactionBytesIngested = metric.Metadata{
		Name:        "rocksdb.compaction.bytes-ingested",

Ingested and compaction are separate concepts. Ingested bytes are not compacted. Compacted bytes are not ingested. I think this should be named rocksdb.ingested-bytes instead. The Help text will also need adjustment. It might be worthwhile chatting with myself or @itsbilal about the difference between ingestion and compaction.


pkg/storage/engine.go, line 486 at r1 (raw file):

	MemtableTotalSize              int64
	Flushes                        int64
	FlushBytes                     int64

Nit: Let's consistently use past tense. This would become FlushedBytes.


pkg/storage/engine.go, line 488 at r1 (raw file):

	FlushBytes                     int64
	Compactions                    int64
	CompactionBytesIngested        int64 // (Pebble only)

Similar to other comment, this should be IngestedBytes.


pkg/storage/engine.go, line 490 at r1 (raw file):

	CompactionBytesIngested        int64 // (Pebble only)
	CompactionBytesRead            int64
	CompactionBytesWritten         int64

Nit: per above: CompactedBytesRead and CompactedBytesWritten.


pkg/ts/catalog/chart_catalog.go, line 1838 at r1 (raw file):

			},
			{
				Title:   "Flushed Bytes",

How do these sectionDescriptions show up in the admin UI? Compaction and flushing are related, so I could see a combined Flush and Compaction section instead of having them separate.

@jbowens jbowens force-pushed the engine-metrics branch 4 times, most recently from df2dd56 to 489fb14 Compare March 26, 2020 14:45
Copy link
Copy Markdown
Contributor Author

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @itsbilal and @petermattis)


pkg/kv/kvserver/metrics.go, line 354 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Ingested and compaction are separate concepts. Ingested bytes are not compacted. Compacted bytes are not ingested. I think this should be named rocksdb.ingested-bytes instead. The Help text will also need adjustment. It might be worthwhile chatting with myself or @itsbilal about the difference between ingestion and compaction.

Done.


pkg/storage/engine.go, line 488 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Similar to other comment, this should be IngestedBytes.

Done.


pkg/ts/catalog/chart_catalog.go, line 1838 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

How do these sectionDescriptions show up in the admin UI? Compaction and flushing are related, so I could see a combined Flush and Compaction section instead of having them separate.

As far as I can tell, these sectionDescriptions are unused. They're surfaced by the /_admin/v1/chartcatalog endpoint, but I wasn't unable to find any consumers of that endpoint. The structure of the graphs under the metrics tab looks like it is specified within the Admin UI's JSX (dashboards/storage.tsx).

There is still a unit test requiring that every metric appear in the chart catalog, which is why I added these here. I'm not sure if that's just a vestige from an older iteration of the Admin UI? The custom chart endpoint also appears to get its list of metrics through a separate /_admin/v1/metricmetadata endpoint, which pulls its data right from the registry, not the catalog.

Copy link
Copy Markdown
Collaborator

@petermattis petermattis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: modulo one final nit to resolve

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @itsbilal and @jbowens)


pkg/kv/kvserver/metrics.go, line 360 at r2 (raw file):

	}
	metaRdbCompactedBytesRead = metric.Metadata{
		Name:        "rocksdb.compaction.bytes-read",

Nit: should this be rocksdb.compacted-bytes-read? That would be more consistent with the variable names and the rocksdb.{flushed,ingested}-bytes metrics. I don't believe the dots signify anything particularly important other than grouping. And not all of the existing compaction metrics have the rocksdb.compaction prefix (e.g. rocksdb.estimated-pending-compaction).


pkg/ts/catalog/chart_catalog.go, line 1838 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

As far as I can tell, these sectionDescriptions are unused. They're surfaced by the /_admin/v1/chartcatalog endpoint, but I wasn't unable to find any consumers of that endpoint. The structure of the graphs under the metrics tab looks like it is specified within the Admin UI's JSX (dashboards/storage.tsx).

There is still a unit test requiring that every metric appear in the chart catalog, which is why I added these here. I'm not sure if that's just a vestige from an older iteration of the Admin UI? The custom chart endpoint also appears to get its list of metrics through a separate /_admin/v1/metricmetadata endpoint, which pulls its data right from the registry, not the catalog.

Ack. Thanks for taking a look.

Copy link
Copy Markdown
Contributor

@itsbilal itsbilal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:LGTM: aside from what Peter said

I imagine we'd have to ditch the rocksdb. prefix for all metrics at some point too, but that can wait for some point in 20.2.

Reviewed 1 of 8 files at r1, 7 of 7 files at r2.
Reviewable status: :shipit: complete! 2 of 0 LGTMs obtained (waiting on @jbowens)

Release justification: low-risk update to new functionality

Fixes cockroachdb#43965

Release note: none
@jbowens
Copy link
Copy Markdown
Contributor Author

jbowens commented Mar 26, 2020

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 26, 2020

Build failed (retrying...)

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 26, 2020

Build succeeded

@craig craig bot merged commit 8fbad6e into cockroachdb:master Mar 26, 2020
@jbowens jbowens deleted the engine-metrics branch March 26, 2020 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

first-pr Use to mark the first PR sent by a contributor / team member. Reviewers should be mindful of this.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage/engine: expose more engine metrics

4 participants