Add Python 3.13 support for Airflow.#46891
Merged
potiuk merged 4 commits intoapache:mainfrom Jul 17, 2025
Merged
Conversation
9888b0f to
f4f2031
Compare
9c6e93c to
b40bf29
Compare
b40bf29 to
21a34cb
Compare
004dd04 to
ec6553d
Compare
Closed
9d382f4 to
25c5d7a
Compare
50fc7ee to
be6b595
Compare
4cf2cc3 to
b9c8b2b
Compare
5e1cd92 to
c85447f
Compare
e39c0f9 to
2d88b64
Compare
jscheffl
reviewed
Jul 15, 2025
jscheffl
approved these changes
Jul 15, 2025
Contributor
jscheffl
left a comment
There was a problem hiding this comment.
Another review round. LIKE It. Special thanks for also adding notes to the selective areas where upper version binding is made. Hope we remember this in future.
Also I very much like the proposal to keep the non-versioned reference image with 3.12 and upgrade as soon as standard installed providers are all Python 3.13 capable. Especially having reference image w/o Fab would make it hardly usable and would confuse users.
I would be happy to get this merged.
Thanks for the efforts.
jscheffl
reviewed
Jul 15, 2025
vincbeck
reviewed
Jul 15, 2025
airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_plugins.py
Outdated
Show resolved
Hide resolved
amoghrajesh
reviewed
Jul 16, 2025
Contributor
amoghrajesh
left a comment
There was a problem hiding this comment.
Some qns / nits on commit 2
amoghrajesh
reviewed
Jul 16, 2025
airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py
Show resolved
Hide resolved
aritra24
approved these changes
Jul 16, 2025
Collaborator
aritra24
left a comment
There was a problem hiding this comment.
This is great Jarek! Had a minor nit, but other than that all looks good from a high level.
Member
Author
|
Pushed changes trying out the comments in places I am not sure. |
aritra24
approved these changes
Jul 17, 2025
gopidesupavan
approved these changes
Jul 17, 2025
Member
gopidesupavan
left a comment
There was a problem hiding this comment.
Thanks @potiuk for all the work, may iterations not an easy task .
Member
|
#protm |
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific behaviors.
* Removal or skipping of tests and features not compatible with Python 3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when runnin tests in Python 3.13
* migration tests had to be excluded fo Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
* Connection is now flushed in `configure_git_connection_for_dag_bundle`
just before `clear_db_connections` - in order to handle Python 3.13
case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
* Cleaning db connections was added to test_dag_run and test_dags test
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.
* Conditionally skipping some serialization tests involving yandex
and ray on Python 3.13 until they support Python 3.13
2 tasks
Member
Author
|
Aaand merged :) |
4 tasks
Collaborator
|
Long effort!! Great job! |
Contributor
|
#protm for sure, great job Jarek! :) |
Contributor
|
🎉 🎉🎉 |
Contributor
|
Wohoo! Party! |
Contributor
|
Cool! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added Python 3.13 support across the codebase, including:
libraries (e.g., numpy 2.x, greenlet, pendulum).
on Python 3.13.
pyarrow, ray, etc.).
dependencies for Python 3.13.
providers.
those depending on Flask AppBuilder).
by Python 3.13.
older FAB provider versions (<1.3.0)
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
is used automatically when runnin tests in Python 3.13
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
configure_git_connection_for_dag_bundlejust before
clear_db_connections- in order to handle Python 3.13case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.