/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

/prombench cancel

Benchmark cancel is in progress.

Looks like stale series tracking is not working. Stale samples are not being put for the series I guess.

/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

/prombench stop

Incorrect /prombench syntax; command requires one argument that matches (master|main|v[0-9]+\.[0-9]+\.[0-9]+\S*) regex.

Available Commands:

• To start benchmark: /prombench <branch or git tag to compare with>
• To restart benchmark: /prombench <branch or git tag to compare with>
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Advanced Flags for start and restart Commands:

• --bench.directory=<sub-directory of github.com/prometheus/test-infra/prombench
  • See the details here, defaults to manifests/prombench.
• --bench.version=<branch | @commit>
  • See the details here, defaults to master.

Examples:

• /prombench v3.0.0
• /prombench v3.0.0 --bench.version=@aca1803ccf5d795eee4b0848707eab26d05965cc --bench.directory=manifests/prombench

/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

/prombench cancel

Benchmark cancel is in progress.

Here are the results from the PoC and the results are in-line with the expectations.

The good: memory and number of timeseries stayed down. For the prombench data pattern, the memory used was consistently 30-60% down.

The bad: The CPU usage is higher and instant queries took a hit. This is mostly because now it requires merging stale block and head block results for instant queries. 20-30% higher for this benchmark and peaks up to 50% sometimes.

I will fix some of the edge cases in the code and run it in our internal cluster to get some real world numbers.

/prombench cancel

Benchmark cancel is in progress.

Prombench decided to keep running even after stopping. So we have more numbers to look at.

Memory savings is great, but query/cpu is bonkers. I am guessing the queries in prombench are trying to touch a whole lot of series continuously.

Took a quick look at the profiles and confirmed that instant queries is taking the extra CPU. In the below pic, the red box is an additional CPU that stale series compaction introduces, since now it has to look at the block on disk for all instant queries. There is no way around it.

Here are the profiles that I downloaded from prombench:
main branch profile.pb.gz
stale series profile.pb.gz

The memory results look really good. For sure something we will want behind a feature flag for now. If we can improve on the CPU overhead, this may be something to enable by default in the future.

I'm surprised instant queries (or any queries) against TSDB blocks are so CPU intensive on Prometheus. OR... prombench results are not realistic -- like it spams queries way to often then realistically users would do. One important case is ofc alerting/recording rules - if they hit TSDB block, that block should be partially cached then (see below)

It would be useful to understand a single common query CPU overhead for in-mem and TSDB block... also we could cache index a bit at least for stale near-real time blocks to mitigate some CPU with a bit more memory (hopefully this will diminishes the memory results - this cache should only be short-living for similar queries or heavy instant query load 🤔). Maybe we learn about some need for optimizing the TSDB read path with this work (:

I still think this would be an interesting mode e.g. for us (Google), where we keep local query capability for debugging in some cases but we use cloud as a first order.

Thanks for extensive research!

prombench results are not realistic -- like it spams queries way to often then realistically users would do

It queries frequently, which might be taken to simulate a large user population or a lot of recording rules, but perhaps more importantly it never queries more than 1 hour back. That is what PRs like prometheus/test-infra#782 are seeking to change.

So if you make every query hit every block, that will make quite a difference.

Hello from the bug-scrub!

@jesusvazquez I see you were assigned - do you think you will get a chance to look at it?

I'll have a look at this next week, starting my PTO today for a few days 🙏

@jesusvazquez I have synced this PR with main branch and fixed the lint and is ready for review

Should we document any recommendations or wait for more user feedback?

We should wait for some user feedback. IMO it's more to do with the pattern in which stale series ratio goes up and down and the memory headroom, and less about the actual value of the ratio. As part of my upcoming talk, I plan to do some more testing for config options.