Skip to content

Add workers.celery.nodeSelector & workers.kubernetes.nodeSelector#61957

Merged
jscheffl merged 2 commits into
apache:mainfrom
Miretpl:add-workers-celery-k8s-node-selector
Mar 12, 2026
Merged

Add workers.celery.nodeSelector & workers.kubernetes.nodeSelector#61957
jscheffl merged 2 commits into
apache:mainfrom
Miretpl:add-workers-celery-k8s-node-selector

Conversation

@Miretpl

@Miretpl Miretpl commented Feb 15, 2026

Copy link
Copy Markdown
Contributor

related: #28880

This PR introduces two new fields: workers.celery.nodeSelector and workers.kubernetes.nodeSelector. The workers.nodeSelector field is not deprecated, as it is used by two worker types. Now it will be possible to use dedicated commands for both Celery and Kubernetes workers.

As an additional change, I split test cases for affinity, tolerations, topology spread constraints and node selectors on global and local level.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@Miretpl Miretpl force-pushed the add-workers-celery-k8s-node-selector branch from c354d6f to b792837 Compare March 12, 2026 15:52
@Miretpl Miretpl marked this pull request as ready for review March 12, 2026 17:23

@jscheffl jscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No deprecation notice in NOTES.txt?

@Miretpl

Miretpl commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

Despite workers.resources, we didn't really deprecated the sections used by both celery and kubernetes, as it would make it easier to define a common configuration for both (having the same resources section for both wouldn't make sense). I'm wondering if we should deprecate all workers sections in favour of workers.celery/workers.kubernetes, but for now, I'm more in favour of no than yes, but I think it will be a part of the discussion about what to deprecate.

For now, as we don't have a concrete decision on what to deprecate, I made it in the same way as before.

@jscheffl

Copy link
Copy Markdown
Contributor

Okay, I understood the general approach was actually to move all parameters into the specific workers.celery/workers.kubernetes sections. I am not sure if in general same resources really apply for both executor types. But keeping multiple parameters in multiple sections in my view render a lot of complexity of respecting two locations. Would rather propose to make it lean and clear.

@Miretpl

Miretpl commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

Okay, I understood the general approach was actually to move all parameters into the specific workers.celery/workers.kubernetes sections

At the beginning, I think the general approach was to make this common parachute over the celery/kubernetes (as it was somewhere in the related issue), and at the beginning, it didn't add complexity to the chart really. The situation changed dramatically after the Worker Set feature, which made everything much more complex.

But keeping multiple parameters in multiple sections in my view render a lot of complexity of respecting two locations.

I agree that having a common parachute will be more complex from the template perspective, and also, after all, the chart user can just duplicate the value between places if there is a need for the same config for both worker types. At the end, having this complexity might not be worth the effort.

Would rather propose to make it lean and clear.

Agreed. I will make a separate PR for already merged sections with the deprecation of them. Regarding this PR, I can include deprecation in the follow-up or add it here for nodeSelector tomorrow/or during the weekend, so feel free to merge/not merge it in the current state

@jscheffl

Copy link
Copy Markdown
Contributor

Merging and until next week when we plan to cus the release there is time to add on-top :-D And if something should be added to 1.20 le me know and I can delay a bit. Plan is to have next release cut starting next week

@jscheffl jscheffl merged commit 025ac8d into apache:main Mar 12, 2026
187 checks passed
@Miretpl

Miretpl commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

I guess it depends on how many releases we want to have before 2.0, as I think that moving the rest of the workers to workers.kubernetes/workers.celery will take around a month more looking at the current speed of it (I already prepared most of the PRs, but I'm also testing one by one to make sure that everything is working correctly). So, I think if we have 1.20, 1.21 and then 2.0, we should be good without the need for delaying the 1.20 release

@Miretpl Miretpl deleted the add-workers-celery-k8s-node-selector branch March 12, 2026 22:07
Pyasma pushed a commit to Pyasma/airflow that referenced this pull request Mar 13, 2026
…ache#61957)

* Split some podTemplate & workers test cases

* Add workers.kubernetes.nodeSelector & workers.celery.nodeSelector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants