Skip to content

ci: run benchmarks on free-threaded build#5760

Merged
davidhewitt merged 4 commits intoPyO3:mainfrom
davidhewitt:benches-3.14t
Jan 30, 2026
Merged

ci: run benchmarks on free-threaded build#5760
davidhewitt merged 4 commits intoPyO3:mainfrom
davidhewitt:benches-3.14t

Conversation

@davidhewitt
Copy link
Copy Markdown
Member

@davidhewitt davidhewitt commented Jan 30, 2026

Given that:

  • users looking for maximum performance from Python are probably going to be shooting for the free-threaded build to make use of parallelism
  • the direction Python seems to be heading is that the free-threaded build is eventually going to be the default build of Python

Maybe it makes sense to run the benchmarks on the free-threaded build?

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Jan 30, 2026

CodSpeed Performance Report

Merging this PR will degrade performance by 78.14%

Comparing davidhewitt:benches-3.14t (b993226) with main (f9b81d4)

Summary

⚡ 51 improved benchmarks
❌ 23 regressed benchmarks
✅ 25 untouched benchmarks
⏩ 1 skipped benchmark1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ dirty_attach 5.1 µs 3.6 µs +41.88%
⚡ test_empty_class_init 24.9 µs 20 µs +24.27%
⚡ clean_attach 3.6 µs 2 µs +80.29%
❌ tuple_nth_back 4 µs 5.9 µs -32.47%
⚡ test_class_attribute 16.1 µs 12.2 µs +32.2%
❌ collect_generic_iterator 98.1 ms 113.8 ms -13.77%
⚡ test_empty_class_init_py 15 µs 12.1 µs +24.65%
⚡ byte_slice_into_pyobject_medium 1.6 µs 1.5 µs +10.41%
❌ tuple_to_list 1.7 ms 1.9 ms -14.94%
⚡ identify_object_type 16.6 µs 14.5 µs +14.72%
❌ iter_list 6.1 ms 10.9 ms -44.02%
⚡ bytes_new_medium 1.6 µs 1.5 µs +10.62%
⚡ test_class_method 18.6 µs 14.3 µs +29.4%
⚡ test_method_call 17 µs 13.3 µs +27.33%
❌ list_get_item 4 ms 4.7 ms -14.57%
⚡ test_method_call_py 15.3 µs 11.6 µs +31.87%
❌ list_nth_back 4 µs 8.6 µs -52.97%
⚡ test_proto_call_py 15.4 µs 11.6 µs +33.02%
⚡ test_static_method 17.7 µs 13.7 µs +29.07%
⚡ list_new 7.3 ms 3.7 ms +96.19%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

@davidhewitt davidhewitt marked this pull request as ready for review January 30, 2026 14:24
@davidhewitt
Copy link
Copy Markdown
Member Author

davidhewitt commented Jan 30, 2026

What do folks think of making this switch? It seems correct to me that we measure on the free-threaded, for the reasons above.

cc @ngoldbaum

@ngoldbaum
Copy link
Copy Markdown
Contributor

ngoldbaum commented Jan 30, 2026

Seems fine to me. Even better to do both, probably - is having two benchmark runs onerous?

@davidhewitt
Copy link
Copy Markdown
Member Author

davidhewitt commented Jan 30, 2026

I think due to limitations in codspeed (can't merge benchmark runs) it's probably one or the other.

@davidhewitt davidhewitt added this pull request to the merge queue Jan 30, 2026
Merged via the queue into PyO3:main with commit 8848c76 Jan 30, 2026
43 of 44 checks passed
@davidhewitt davidhewitt deleted the benches-3.14t branch January 30, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@davidhewitt @ngoldbaum