/prombench main --bench.version=bench/cross-feature/ct-per-sample

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16046 and main

Custom benchmark version: bench/cross-feature/ct-per-sample branch

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main --bench.version=bench/cross-feature/ct-per-sample
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Functionality

Ok, it seems working for majority of incoming cumulatives. Not for native histograms so this is why there are still ones without CTs:

Efficiency

In terms of perf (for samples), we see some overhead:

A lot more allocs/s

Slight CPU increase

Up to 115ms vs 80ms latency spikes on query_range:

Slight increase in RSS:

Weird stuff:

I see odd notification/live long API call, not present on main...

pprof

• PR
• main

Huh, so based on the profiles majority of allocated objects (other than querying) comes from protobuf parser still (magic labels) and CreatedTimestamp validation (and error?).

No error logs on Prometheus though.

/prombench cancel

Benchmark cancel is in progress.

No regression in terms of memory on query side. Majority of allocs comes from proto parsing still. We might want to change benchmark scenario to include proto parsing for both...

Decoding is heavy though (from WAL watcher), which is 2x heavier for some reason:

vs main

... oh I think we have to update our calculation here:

In terms of objects

RW package changes look fine 👍

Re: the allocations issue, you're suggesting that the min size needs to account for the additional integer?
dec.Len() / (1 + 1 + 1 + 8); cap(samples) < minSize?

It feels so, yea. I want to repeat the experiment with both versions doing protobuf though for a fair experiment.

Next steps:

• Redo benchmark with both using protobuf parsing, double check interesting query timing results.
• Review this PR, especially how I decided to form this flag.
• Connect with people who are working on OTLP detlas in Prometheus, because I think they 1:1 use this feature cc @ArthurSens -- especially if we go this route with detlas, is there anything worth changing in the struct/WAL format while we are at it. For example, we could consider renaming "Created Timestamp" fields into something like "Start time"? although we pushed for created timestamp naming already and maybe it's fine.

Retrying benchmark after optimizations in watcher. Custom scenario have one update too -- proto parsing is now enabled on both versions of Prometheus.

/prombench restart main --bench.version=bench/cross-feature/ct-per-sample

/prombench start main --bench.version=bench/cross-feature/ct-per-sample

Incorrect /prombench syntax; command requires one argument that matches (master|main|v[0-9]+\.[0-9]+\.[0-9]+\S*) regex.

Available Commands:

• To start benchmark: /prombench <branch or git tag to compare with>
• To restart benchmark: /prombench <branch or git tag to compare with>
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Advanced Flags for start and restart Commands:

• --bench.directory=<sub-directory of github.com/prometheus/test-infra/prombench
  • See the details here, defaults to manifests/prombench.
• --bench.version=<branch | @commit>
  • See the details here, defaults to master.

Examples:

• /prombench v3.0.0
• /prombench v3.0.0 --bench.version=@aca1803ccf5d795eee4b0848707eab26d05965cc --bench.directory=manifests/prombench

/prombench main --bench.version=bench/cross-feature/ct-per-sample

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16046 and main

Custom benchmark version: bench/cross-feature/ct-per-sample branch

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main --bench.version=bench/cross-feature/ct-per-sample
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Thanks to decoder/watcher fixes I see significant improvement in RSS and allocs/s vs main WITHOUT CTs in WAL 😱

CPU etc looks on par

Query engine timing is worse - not sure why:

CPU wise it looks like GC is more intensive, querying is actually using more CPU on main 🤔 https://pprof.me/33106dee65aec562860070b28ed2d895

Looks like simply more objects are allocated https://pprof.me/c23cf4fda6e7be3ca34d292a30fc5f13
Total memory is seems better on this PR: https://pprof.me/0c232207d6fd0f7e461054bb972ebbb0

/prombench cancel

Benchmark cancel is in progress.

Adding a few PRs first to split this change into smaller parts.

e.g. #16072, #16156, #16182

TODO:

• Rebase it on main and use type-and-unit feature
• Double check if we need wlog: Optimized and refactored watcher code. #16182 or can we pursue this in separate PR.
• Ensure type and unit is constructing PRWv2 (as per [meta] PROM-39 type-and-unit-labels stability #16610) (separate PR)
• Benchmark and find consensus on the WAL format (e.g. based on https://github.com/prometheus/test-infra/tree/bench/cross-feature/ct-per-sample/prombench/manifests/prombench but instead of metada-wal-record, use type-and-unit feature)

The main non-trivial decision is about that WAL format. In this PR I propose CT next to every sample value and timestamp (all is diff-encoded) and it was showing not that big overhead, but we have to convince others.

The alternative is to perhaps design something similar to native histograms records, so create a record per different CT.

cc @ridwanmsharif