Skip to content

Improve benchmarking command line #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mingxwa merged 2 commits into from
Oct 14, 2024
Merged

Improve benchmarking command line #177

mingxwa merged 2 commits into from
Oct 14, 2024

Conversation

@mingxwa
Copy link
Member

@mingxwa mingxwa commented Oct 14, 2024 •
edited
Loading

After the update, the result number becomes more stable.

Raw data from CI build:

MSVC on Windows Server 2022 (x64) GCC on Ubuntu 24.04 (x64) Clang on Ubuntu 24.04 (x64) Apple Clang on macOS 15 (ARM64)
Indirect invocation on small objects via proxy 9,754,272 ns 8,816,234 ns 8,442,648 ns 2,504,394 ns
Indirect invocation on small objects via virtual functions 34,842,900 ns 11,528,170 ns 11,614,613 ns 2,630,067 ns
Indirect invocation on large objects via proxy 14,117,789 ns 11,189,139 ns 11,605,143 ns 4,926,853 ns
Indirect invocation on large objects via virtual functions 53,516,325 ns 14,540,389 ns 13,436,771 ns 5,010,965 ns
Basic lifetime management for small objects with proxy 1,260,437 ns 656,388 ns 682,870 ns 568,324 ns
Basic lifetime management for small objects with std::unique_ptr 5,832,715 ns 3,271,340 ns 3,297,287 ns 2,231,709 ns
Basic lifetime management for small objects with std::shared_ptr (without memory pool) 7,356,122 ns 3,801,466 ns 3,659,358 ns 2,913,971 ns
Basic lifetime management for small objects with std::shared_ptr (with memory pool) 4,113,066 ns 4,515,817 ns 4,505,196 ns 1,629,097 ns
Basic lifetime management for small objects with std::any 1,890,091 ns 2,456,667 ns 2,518,811 ns 667,104 ns
Basic lifetime management for large objects with proxy (without memory pool) 8,775,308 ns 10,891,469 ns 10,462,914 ns 3,841,652 ns
Basic lifetime management for large objects with proxy (with memory pool) 4,126,878 ns 5,940,603 ns 4,631,703 ns 1,858,271 ns
Basic lifetime management for large objects with std::unique_ptr 8,428,636 ns 11,776,444 ns 10,356,230 ns 3,504,924 ns
Basic lifetime management for large objects with std::shared_ptr (both without memory pool) 9,835,728 ns 12,520,650 ns 11,313,639 ns 4,351,325 ns
Basic lifetime management for large objects with std::shared_ptr (both with memory pool) 4,526,459 ns 6,948,993 ns 5,015,226 ns 2,712,781 ns

Comparison:

MSVC on Windows Server 2022 (x64) GCC on Ubuntu 24.04 (x64) Clang on Ubuntu 24.04 (x64) Apple Clang on macOS 15 (ARM64)
Indirect invocation on small objects via proxy vs. virtual functions 🟩 proxy is about 257.2% faster 🟩 proxy is about 30.8% faster 🟩 proxy is about 37.6% faster 🟩 proxy is about 5.0% faster
Indirect invocation on large objects via proxy vs. virtual functions 🟩 proxy is about 279.1% faster 🟩 proxy is about 30.0% faster 🟩 proxy is about 15.8% faster 🟨 proxy is about 1.7% faster
Basic lifetime management for small objects with proxy vs. std::unique_ptr 🟩 proxy is about 362.8% faster 🟩 proxy is about 398.4% faster 🟩 proxy is about 382.9% faster 🟩 proxy is about 292.7% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (without memory pool) 🟩 proxy is about 483.6% faster 🟩 proxy is about 479.1% faster 🟩 proxy is about 435.9% faster 🟩 proxy is about 412.7% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (with memory pool) 🟩 proxy is about 226.3% faster 🟩 proxy is about 588.0% faster 🟩 proxy is about 559.7% faster 🟩 proxy is about 186.6% faster
Basic lifetime management for small objects with proxy vs. std::any 🟩 proxy is about 50.0% faster 🟩 proxy is about 274.3% faster 🟩 proxy is about 268.9% faster 🟩 proxy is about 17.4% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::unique_ptr 🟨 proxy is about 4.0% slower 🟩 proxy is about 8.1% faster 🟨 proxy is about 1.0% slower 🟥 proxy is about 8.8% slower
Basic lifetime management for large objects with proxy (with memory pool) vs. std::unique_ptr 🟩 proxy is about 104.2% faster 🟩 proxy is about 98.2% faster 🟩 proxy is about 123.6% faster 🟩 proxy is about 88.6% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both without memory pool) 🟩 proxy is about 12.1% faster 🟩 proxy is about 15.0% faster 🟩 proxy is about 8.1% faster 🟩 proxy is about 13.3% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both with memory pool) 🟩 proxy is about 9.7% faster 🟩 proxy is about 17.0% faster 🟩 proxy is about 8.3% faster 🟩 proxy is about 46.0% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::any 🟩 proxy is about 27.1% faster 🟨 proxy is about 0.9% faster 🟨 proxy is about 0.3% slower 🟨 proxy is about 4.2% faster
Basic lifetime management for large objects with proxy (with memory pool) vs. std::any 🟩 proxy is about 170.2% faster 🟩 proxy is about 85.1% faster 🟩 proxy is about 125.3% faster 🟩 proxy is about 115.3% faster

@mingxwa mingxwa requested review from guominrui and tian-lt October 14, 2024 09:18
@mingxwa mingxwa merged commit 2d776ea into microsoft:main Oct 14, 2024
@mingxwa mingxwa deleted the user/mingxwa/benchmark-script-args branch October 14, 2024 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tian-lt tian-lt tian-lt approved these changes

@guominrui guominrui Awaiting requested review from guominrui

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mingxwa @tian-lt