GH-39303: [Archery][Benchmarking] Allow setting C++ repetition min time#39324
GH-39303: [Archery][Benchmarking] Allow setting C++ repetition min time#39324kou merged 2 commits intoapache:mainfrom
Conversation
…min time We want to be able to increase the number of repetitions for each C++ micro-benchmark without increase the total runtime.
|
With this PR, if I run: $ archery benchmark run /build/build-release/ --suite-filter arrow-value-parsing-benchmark --benchmark-filter "FloatParsing.DoubleType" --repetitions 20 --repetition-min-time=0.05 --output=t.json... then the total runtime is less than 2 second and I get a results file like this: {
"suites": [
{
"name": "arrow-value-parsing-benchmark",
"benchmarks": [
{
"name": "FloatParsing<DoubleType>",
"unit": "items_per_second",
"less_is_better": false,
"values": [
79950121.48431177,
85113591.18022932,
90551140.90867223,
90609487.2652054,
90628541.38461946,
90649225.23022957,
91166686.43369755,
91220387.63998744,
91538368.41832638,
91640734.18134348,
91846292.52809268,
91929727.83451708,
91948825.97105175,
91975895.10168795,
92096737.69094324,
92171049.58800516,
92262171.0999602,
92316911.2854944,
92499358.69477023,
93462194.03456633
],
"time_unit": "ns",
"times": [
10717.94139001506,
10813.586956610783,
10835.81713547928,
10852.307757681898,
10854.492114290344,
10871.693946617952,
10876.263213883914,
10879.553708338452,
10881.460784591169,
10905.518542276943,
10916.3064791786,
10929.045609416553,
10967.386402201393,
10972.461636954646,
11034.734867894433,
11037.859122534785,
11046.999999422633,
11047.337169190347,
11760.27770670678,
12512.818840690854
],
"counters": {
"family_index": 0,
"per_family_instance_index": 0,
"run_name": "FloatParsing<DoubleType>",
"repetitions": 20,
"repetition_index": 9,
"threads": 1,
"iterations": 4692
}
}
]
}
]
} |
9f89e51 to
2cd0f0c
Compare
That clarity of wording is much appreciated.
Lovely. For people reading along, a related thread about that terminology: conbench/conbench#1398 |
| @@ -0,0 +1,2 @@ | |||
| pytest | |||
There was a problem hiding this comment.
Well, apparently requirements files are excluded from the RAT checks?
I presume comments are not supported in those files.
There was a problem hiding this comment.
Well, apparently requirements files are excluded from the RAT checks?
In general, I think that we should use the RAT checks as much as possible for safety.
I presume comments are not supported in those files.
https://github.com/apache/arrow/blob/main/docs/requirements.txt uses comment. So we may be able to use comment here too. If we can't, I'm OK with excluding from the RAT checks.
We can defer this to another PR if needed.
I'll merge this.
|
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 1c1fa74. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them. |
…min time (apache#39324) ### Rationale for this change We want to be able to increase the number of repetitions for each C++ micro-benchmark without increasing the total runtime. ### What changes are included in this PR? * Add a `--repetition-min-time` argument to set the repetition duration in seconds * Add a `--cpp-benchmark-extras` argument to pass arbitrary arguments to Google Benchmark executables * Add a couple tests with multiple benchmark repetitions ### Are these changes tested? Not entirely. Command-line argument passing is not unit-tested. ### Are there any user-facing changes? No. * Closes: apache#39303 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Rationale for this change
We want to be able to increase the number of repetitions for each C++ micro-benchmark without increasing the total runtime.
What changes are included in this PR?
--repetition-min-timeargument to set the repetition duration in seconds--cpp-benchmark-extrasargument to pass arbitrary arguments to Google Benchmark executablesAre these changes tested?
Not entirely. Command-line argument passing is not unit-tested.
Are there any user-facing changes?
No.