Use Cython 3.1 universally by lysnikolaou · Pull Request #1514 · aio-libs/yarl

lysnikolaou · 2025-06-03T12:10:43Z

What do these changes do?

Use Cython 3.1 universally
Delete unnecessary requirement files

Are there changes in behavior for the user?

No.

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes

codspeed-hq · 2025-06-03T12:25:42Z

CodSpeed Performance Report

Merging #1514 will degrade performances by 57.73%

_{Comparing lysnikolaou:cython-31 (4fd60e7) with master (63c4461)}

Summary

❌ 34 (👁 34) regressions
✅ 67 untouched benchmarks

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
👁	`test_long_query`	5.9 ms	13.2 ms	-55.62%
👁	`test_long_query_with_pct`	17.9 ms	41.2 ms	-56.47%
👁	`test_quote_long_path`	4.8 ms	10.9 ms	-55.56%
👁	`test_quote_query_string`	314.2 µs	627.3 µs	-49.91%
👁	`test_quoter_ascii`	124.2 µs	204 µs	-39.12%
👁	`test_quoter_pct`	245.6 µs	483.3 µs	-49.18%
👁	`test_quoter_quote_utf8`	684.7 µs	1,619.9 µs	-57.73%
👁	`test_unquoter_short`	204 µs	225 µs	-9.33%
👁	`test_extend_query_subclassed_str`	1 ms	1.2 ms	-12.66%
👁	`test_parse_query_uncached[long]`	21 ms	23.4 ms	-10.23%
👁	`test_parse_query_uncached[short]`	1.7 ms	2 ms	-14.8%
👁	`test_path_safe_uncached`	230.9 µs	287.4 µs	-19.67%
👁	`test_update_query_mapping`	942.8 µs	1,081.7 µs	-12.84%
👁	`test_update_query_mapping_with_existing_query`	1.3 ms	1.5 ms	-16.09%
👁	`test_update_query_sequence_mapping`	4.5 ms	5.9 ms	-22.5%
👁	`test_update_query_string`	611.1 µs	660.5 µs	-7.48%
👁	`test_url_build_access_raw_path`	595.4 µs	651.4 µs	-8.6%
👁	`test_url_build_no_netloc`	432.2 µs	543.9 µs	-20.54%
👁	`test_url_build_no_netloc_relative`	419.1 µs	525.4 µs	-20.23%
👁	`test_url_build_with_host_and_port`	640.2 µs	702.7 µs	-8.9%
...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

codecov · 2025-06-03T12:37:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.63%. Comparing base (63c4461) to head (4fd60e7).
⚠️ Report is 44 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1514      +/-   ##
==========================================
+ Coverage   99.62%   99.63%   +0.01%     
==========================================
  Files          29       29              
  Lines        5798     5795       -3     
  Branches      265      265              
==========================================
- Hits         5776     5774       -2     
+ Misses         19       18       -1     
  Partials        3        3

Flag	Coverage Δ
CI-GHA	`99.63% <100.00%> (+0.01%)`	⬆️
MyPy	`98.07% <100.00%> (+0.01%)`	⬆️
OS-Linux	`99.80% <ø> (ø)`
OS-Windows	`99.48% <ø> (ø)`
OS-macOS	`99.68% <ø> (ø)`
Py-3.10.11	`99.45% <ø> (ø)`
Py-3.10.17	`99.70% <ø> (ø)`
Py-3.11.12	`99.70% <ø> (ø)`
Py-3.11.9	`99.45% <ø> (ø)`
Py-3.12.10	`99.70% <ø> (ø)`
Py-3.13.3	`99.75% <ø> (+0.04%)`	⬆️
Py-3.13.4t	`99.75% <ø> (ø)`
Py-3.9.13	`99.41% <ø> (ø)`
Py-3.9.22	`99.65% <ø> (ø)`
Py-pypy7.3.16	`99.33% <ø> (ø)`
Py-pypy7.3.19	`99.36% <ø> (ø)`
VM-macos-latest	`99.68% <ø> (ø)`
VM-ubuntu-latest	`99.80% <ø> (ø)`
VM-windows-latest	`99.48% <ø> (ø)`
pytest	`99.80% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

webknjaz · 2025-06-04T13:00:30Z

@bdraco is this a legit coverage hit?

Copilot

Pull Request Overview

This PR updates the project to use Cython 3.1 universally while removing obsolete requirement files and related conditional dependency handling. Key changes include updating Cython version in requirement files and build backend, deleting test-pure and cython-freethreading requirements, and simplifying the CI/CD configuration.

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
requirements/test.txt	Updated to include additional test dependencies and reference cython.txt
requirements/test-pure.txt	Removed redundant testing requirements
requirements/test-freethreading.txt	Removed obsolete freethreading testing requirements
requirements/cython.txt	Updated Cython version from 3.0.12 to 3.1.1
requirements/cython-freethreading.txt	Removed obsolete Cython freethreading requirement
pyproject.toml	Removed override configuration related to Cython freethreading
packaging/pep517_backend/_backend.py	Updated Cython dependency handling by removing the special branch for GIL-disabled builds
CHANGES/1514.packaging.rst	Documented the universal update to Cython 3.1
.github/workflows/ci-cd.yml	Simplified dependency reference for CI/CD builds

Comments suppressed due to low confidence (2)

packaging/pep517_backend/_backend.py:377

The removal of the dedicated branch for GIL-disabled builds simplifies dependency management, but please confirm that builds relying on the prior behavior are properly handled with Cython 3.1 universally.

elif sysconfig.get_config_var('Py_GIL_DISABLED'):

.github/workflows/ci-cd.yml:262

Changing the dependency specification from a conditional formulation to a fixed 'requirements/test.txt' may affect build configurations that previously used freethreading tests; please verify that these changes align with the intended build strategy.

requirements/test.txt

bdraco · 2025-06-04T13:01:48Z

Will check it when I'm back at my desk

webknjaz

LGTM functionally, but it'd be good to figure out the benchmark thing. I'll let @bdraco decide whether this is mergeable.

bdraco · 2025-06-04T17:16:16Z

At least for some of them it's now including the Cython tracing overhead and the performance regression is not real. It will take me a few hours to investigate the rest of them. I'm at the OHF summit in Dublin this week so I'm quite time limited and trying to only focus on blockers and bugs this week

bdraco · 2025-06-04T17:16:31Z

I will do a deep dive next week when I'm back in Texas

bdraco · 2025-06-09T18:39:18Z

aio-libs/frozenlist#658 seems to imply we have an issue with production wheels using line tracing

packaging/pep517_backend/_backend.py

requirements/cython.txt

bdraco · 2025-06-09T18:57:00Z

#1517

Hit the same way. Let me do a PR with line tracing off and see what we get #1519

bdraco · 2025-06-09T19:14:15Z

Here are the benchmark results with line tracing disabled against the previous release
https://codspeed.io/aio-libs/yarl/runs/compare/680042e6d1ca3d2211b58b12..68473196e1567421a9b96153

The problem is we are benchmarking with line tracing for PRs and no line tracing for releases. We should be benchmarking what we are actually going to release. I will make a PR to fix that.

webknjaz · 2025-06-09T20:36:48Z

@lysnikolaou shouldn't we also use the freethreading_compatible setting, by the way?

bdraco · 2025-06-09T21:02:55Z

So after much investigation it turns out we are shipping wheels with linetrace on

#1521 will fix this.

lysnikolaou · 2025-06-10T10:49:19Z

@lysnikolaou shouldn't we also use the freethreading_compatible setting, by the way?

We're doing that already in the .pyx file itself.

webknjaz · 2025-06-10T14:56:39Z

@lysnikolaou shouldn't we also use the freethreading_compatible setting, by the way?

We're doing that already in the .pyx file itself.

@lysnikolaou oh, I see… I wonder if it'd be a good idea to centralize those settings through https://github.com/aio-libs/yarl/blob/3871d2c/pyproject.toml#L22-L37 so that individual pyx files wouldn't need a copy of that comment.

lysnikolaou · 2025-06-10T18:07:51Z

I wonder if it'd be a good idea to centralize those settings through 3871d2c/pyproject.toml#L22-L37 so that individual pyx files wouldn't need a copy of that comment.

Yeah, that's a good point. This wasn't possible up until now, because we were using different versions of Cython that did not have support for the freethreading_compatible directive, but it's now okay since we upgrade to 3.1. Opened #1528.

Since the python-cython bump to v3.1.2 (see [1]), python-yarl has been failing on the autobuilder with the following error message: ``` ERROR Missing dependencies: Cython~=3.0.0; python_version >= "3.12" make: *** [package/pkg-generic.mk:273: /home/buildroot/instance-0/output-1/build/python-yarl-1.18.3/.stamp_built] Error 1 ``` The cython dependency has been addressed in python-yarl v1.20.1, for more information see the github issue [2]. For more information on the release, see: - https://github.com/aio-libs/yarl/releases/tag/v1.19.0 - https://github.com/aio-libs/yarl/releases/tag/v1.20.0 - https://github.com/aio-libs/yarl/releases/tag/v1.20.1 [1] b536caa package/python-cython: bump to version 3.1.2 [2] aio-libs/yarl#1514 Fixes: https://autobuild.buildroot.org/results/d36/d367b69b85a65fa94e923ecff3ba03723b2a6e88 Signed-off-by: Thomas Perale <thomas.perale@mind.be> Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>

Since the python-cython bump to v3.1.2 (see [1]), python-yarl has been failing on the autobuilder with the following error message: ``` ERROR Missing dependencies: Cython~=3.0.0; python_version >= "3.12" make: *** [package/pkg-generic.mk:273: /home/buildroot/instance-0/output-1/build/python-yarl-1.18.3/.stamp_built] Error 1 ``` The cython dependency has been addressed in python-yarl v1.20.1, for more information see the github issue [2]. For more information on the release, see: - https://github.com/aio-libs/yarl/releases/tag/v1.19.0 - https://github.com/aio-libs/yarl/releases/tag/v1.20.0 - https://github.com/aio-libs/yarl/releases/tag/v1.20.1 [1] b536caa package/python-cython: bump to version 3.1.2 [2] aio-libs/yarl#1514 Fixes: https://autobuild.buildroot.org/results/d36/d367b69b85a65fa94e923ecff3ba03723b2a6e88 Signed-off-by: Thomas Perale <thomas.perale@mind.be> Signed-off-by: Arnout Vandecappelle <arnout@rnout.be> (cherry picked from commit 0bd8814) Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>

lysnikolaou added 2 commits June 3, 2025 14:10

Use Cython 3.1 universally

d97ad62

Add changelog item

e7e3a5a

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Jun 3, 2025

Fix lint error

5875bf2

webknjaz requested review from bdraco and Copilot June 4, 2025 13:00

Copilot AI reviewed Jun 4, 2025

View reviewed changes

webknjaz approved these changes Jun 4, 2025

View reviewed changes

bdraco reviewed Jun 9, 2025

View reviewed changes

packaging/pep517_backend/_backend.py Outdated Show resolved Hide resolved

Update packaging/pep517_backend/_backend.py

27199ae

bdraco reviewed Jun 9, 2025

View reviewed changes

requirements/cython.txt Outdated Show resolved Hide resolved

Update requirements/cython.txt

4ba2397

Merge branch 'master' into cython-31

4fd60e7

bdraco approved these changes Jun 9, 2025

View reviewed changes

bdraco merged commit 67ce430 into aio-libs:master Jun 9, 2025
52 of 54 checks passed

Uh oh!

Conversation

lysnikolaou commented Jun 3, 2025

What do these changes do?

Are there changes in behavior for the user?

Checklist

Uh oh!

codspeed-hq bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #1514 will degrade performances by 57.73%

Summary

Benchmarks breakdown

Uh oh!

codecov bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

webknjaz commented Jun 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

bdraco commented Jun 4, 2025

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

bdraco commented Jun 4, 2025

Uh oh!

bdraco commented Jun 4, 2025

Uh oh!

bdraco commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

bdraco commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdraco commented Jun 9, 2025

Uh oh!

webknjaz commented Jun 9, 2025

Uh oh!

bdraco commented Jun 9, 2025

Uh oh!

Uh oh!

lysnikolaou commented Jun 10, 2025

Uh oh!

webknjaz commented Jun 10, 2025

Uh oh!

lysnikolaou commented Jun 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codspeed-hq bot commented Jun 3, 2025 •

edited

Loading

codecov bot commented Jun 3, 2025 •

edited

Loading

bdraco commented Jun 9, 2025 •

edited

Loading