Skip to content

Stabilize benchmarking report #183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mingxwa merged 2 commits into from
Oct 28, 2024
Merged

Stabilize benchmarking report #183

mingxwa merged 2 commits into from
Oct 28, 2024

Conversation

@mingxwa
Copy link
Member

@mingxwa mingxwa commented Oct 27, 2024 •
edited
Loading

Changes

  • Updated benchmarking command line:
    • Added benchmark_min_warmup_time=0.1 to warmup each benchmark.
    • Changed benchmark_repetitions from 10 to 30 to get more test results.
    • Added --benchmark_min_time=0.1s to keep the overall execution time similar with before.
    • Added benchmark_enable_random_interleaving=true to avoid transient high load.
  • For invocation benchmarks BM_SmallObjectInvocationViaProxy, BM_SmallObjectInvocationViaVirtualFunction, BM_LargeObjectInvocationViaProxy and BM_LargeObjectInvocationViaVirtualFunction, moved data preparation from global to each function body to make sure each case runs from a clean state.
  • Changed padding of large objects from 15 * sizeof(void*) into 5 * sizeof(void*) to have less effects on heap allocation.
  • For lifetime management benchmarks, changed test object count from 12,000 into 30,000 for better differentiation.
  • Enabled workflow_dispatch for CI pipeline to facilitate benchmarking.

Here's the report generated from the CI build of this PR.

Benchmarking Report

MSVC on Windows Server 2022 (x64) GCC on Ubuntu 24.04 (x64) Clang on Ubuntu 24.04 (x64) Apple Clang on macOS 15 (ARM64)
Indirect invocation on small objects via proxy vs. virtual functions 🟢proxy is about 233.7% faster 🟢proxy is about 39.5% faster 🟢proxy is about 44.7% faster 🟢proxy is about 6.7% faster
Indirect invocation on large objects via proxy vs. virtual functions 🟢proxy is about 173.2% faster 🟢proxy is about 16.2% faster 🟢proxy is about 14.8% faster 🟢proxy is about 10.2% faster
Basic lifetime management for small objects with proxy vs. std::unique_ptr 🟢proxy is about 439.1% faster 🟢proxy is about 87.2% faster 🟢proxy is about 378.3% faster 🟢proxy is about 302.0% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (without memory pool) 🟢proxy is about 672.9% faster 🟢proxy is about 115.4% faster 🟢proxy is about 492.3% faster 🟢proxy is about 438.8% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (with memory pool) 🟢proxy is about 187.4% faster 🟢proxy is about 181.4% faster 🟢proxy is about 652.7% faster 🟢proxy is about 156.1% faster
Basic lifetime management for small objects with proxy vs. std::any 🟢proxy is about 56.2% faster 🟢proxy is about 47.7% faster 🟢proxy is about 323.8% faster 🟢proxy is about 14.1% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::unique_ptr 🟢proxy is about 18.7% faster 🟢proxy is about 8.1% faster 🟢proxy is about 11.7% faster 🔴proxy is about 5.6% slower
Basic lifetime management for large objects with proxy (with memory pool) vs. std::unique_ptr 🟢proxy is about 265.8% faster 🟢proxy is about 90.7% faster 🟢proxy is about 130.8% faster 🟢proxy is about 93.1% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both without memory pool) 🟢proxy is about 34.1% faster 🟢proxy is about 5.3% faster 🟢proxy is about 7.6% faster 🟢proxy is about 5.6% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both with memory pool) 🟢proxy is about 8.9% faster 🟢proxy is about 9.4% faster 🟢proxy is about 16.7% faster 🟢proxy is about 42.9% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::any 🟢proxy is about 15.3% faster 🟡proxy is about 3.1% slower 🟡proxy is about 1.0% faster 🟢proxy is about 5.9% faster
Basic lifetime management for large objects with proxy (with memory pool) vs. std::any 🟢proxy is about 255.5% faster 🟢proxy is about 71.0% faster 🟢proxy is about 108.6% faster 🟢proxy is about 116.5% faster

@mingxwa mingxwa requested review from guominrui and tian-lt October 27, 2024 09:14
@mingxwa mingxwa marked this pull request as draft October 27, 2024 10:05
@mingxwa mingxwa marked this pull request as ready for review October 28, 2024 01:24
@mingxwa mingxwa merged commit d742a33 into microsoft:main Oct 28, 2024
@mingxwa mingxwa deleted the user/mingxwa/benchmark-stablization branch October 28, 2024 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tian-lt tian-lt tian-lt approved these changes

@guominrui guominrui guominrui approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mingxwa @tian-lt @guominrui