Add workers.celery.nodeSelector & workers.kubernetes.nodeSelector#61957
Conversation
c354d6f to
b792837
Compare
jscheffl
left a comment
There was a problem hiding this comment.
No deprecation notice in NOTES.txt?
|
Despite For now, as we don't have a concrete decision on what to deprecate, I made it in the same way as before. |
|
Okay, I understood the general approach was actually to move all parameters into the specific workers.celery/workers.kubernetes sections. I am not sure if in general same resources really apply for both executor types. But keeping multiple parameters in multiple sections in my view render a lot of complexity of respecting two locations. Would rather propose to make it lean and clear. |
At the beginning, I think the general approach was to make this common parachute over the celery/kubernetes (as it was somewhere in the related issue), and at the beginning, it didn't add complexity to the chart really. The situation changed dramatically after the Worker Set feature, which made everything much more complex.
I agree that having a common parachute will be more complex from the template perspective, and also, after all, the chart user can just duplicate the value between places if there is a need for the same config for both worker types. At the end, having this complexity might not be worth the effort.
Agreed. I will make a separate PR for already merged sections with the deprecation of them. Regarding this PR, I can include deprecation in the follow-up or add it here for nodeSelector tomorrow/or during the weekend, so feel free to merge/not merge it in the current state |
|
Merging and until next week when we plan to cus the release there is time to add on-top :-D And if something should be added to 1.20 le me know and I can delay a bit. Plan is to have next release cut starting next week |
|
I guess it depends on how many releases we want to have before 2.0, as I think that moving the rest of the |
…ache#61957) * Split some podTemplate & workers test cases * Add workers.kubernetes.nodeSelector & workers.celery.nodeSelector
related: #28880
This PR introduces two new fields:
workers.celery.nodeSelectorandworkers.kubernetes.nodeSelector. Theworkers.nodeSelectorfield is not deprecated, as it is used by two worker types. Now it will be possible to use dedicated commands for both Celery and Kubernetes workers.As an additional change, I split test cases for affinity, tolerations, topology spread constraints and node selectors on global and local level.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.