Skip to content

Filter celery stuck task query to exclude completed tasks#21335

Merged
ephraimbuddy merged 2 commits intoapache:mainfrom
astronomer:filter-task
Feb 7, 2022
Merged

Filter celery stuck task query to exclude completed tasks#21335
ephraimbuddy merged 2 commits intoapache:mainfrom
astronomer:filter-task

Conversation

@ephraimbuddy
Copy link
Contributor

On testing #19769, it was reported that there was a spike in CPU usage #19769 (comment)
Hopefully, this will fix it

cc: @jedcunningham


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Feb 4, 2022
@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Feb 4, 2022
@github-actions
Copy link

github-actions bot commented Feb 4, 2022

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After mocking this query, because the update_all_task_states has the same query at BulkStateFetcher, it throws an error. I had to mock the update_all_task_states too

On testing apache#19769, it was reported that there was a spike in CPU usage apache#19769 (comment)
Hopefully, this will fix it
@ephraimbuddy ephraimbuddy added this to the Airflow 2.2.4 milestone Feb 7, 2022
@ephraimbuddy ephraimbuddy merged commit a49224f into apache:main Feb 7, 2022
@ephraimbuddy ephraimbuddy deleted the filter-task branch February 7, 2022 09:43
@ephraimbuddy
Copy link
Contributor Author

Thanks, @jedcunningham for spotting this!

@jedcunningham jedcunningham added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Feb 8, 2022
jedcunningham pushed a commit that referenced this pull request Feb 8, 2022
On testing #19769, it was reported that there was a spike in CPU usage #19769 (comment)
Hopefully, this will fix it

(cherry picked from commit a49224f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants