Simplify logic to resolve tasks stuck in queued despite stalled_task_timeout#30375
Merged
ephraimbuddy merged 34 commits intoapache:mainfrom Apr 14, 2023
Merged
Simplify logic to resolve tasks stuck in queued despite stalled_task_timeout#30375ephraimbuddy merged 34 commits intoapache:mainfrom
ephraimbuddy merged 34 commits intoapache:mainfrom
Conversation
ephraimbuddy
reviewed
Apr 4, 2023
Contributor
ephraimbuddy
left a comment
There was a problem hiding this comment.
Just reviewed only the scheduler_job, will come back again
d000e7a to
03aaf06
Compare
jedcunningham
previously requested changes
Apr 6, 2023
ephraimbuddy
reviewed
Apr 8, 2023
ephraimbuddy
approved these changes
Apr 14, 2023
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
This more closely mirrors how deprecations are raised for "normal" deprecations. I've removed the depth, as moving up the stack doesn't really help the user at all in this situation.
hterik
reviewed
Apr 14, 2023
| for ti in tis: | ||
| readable_tis.append(repr(ti)) | ||
| task_instance_key = ti.key | ||
| self.fail(task_instance_key, None) |
Contributor
There was a problem hiding this comment.
Not sure if the data model allows, but is it possible to add some error details to the taskinstance here so a user can understands why it failed, by just looking in the web ui?
To avoid admins getting asked a lot of questions from users like "why did this task fail without any logs", having to open up scheduler logs.
Contributor
Author
There was a problem hiding this comment.
My understanding is that there's not an easy way to surface these logs in the UI. Such "missing" task logs could also be caused by zombies, which can be caused by... tons of stuff.
I think a good intermediate step will be to add a blurb in the docs about missing task logs for zombies as well as tasks stuck in queued. I plan to open such a docs PR soon-ish.
Contributor
Author
There was a problem hiding this comment.
This will likely be possible in Airflow 2.8.0 with this PR: #32646
wookiist
pushed a commit
to wookiist/airflow
that referenced
this pull request
Apr 19, 2023
…timeout (apache#30375) * simplify and consolidate logic for tasks stuck in queued * simplify and consolidate logic for tasks stuck in queued * simplify and consolidate logic for tasks stuck in queued * fixed tests; updated fail stuck tasks to use run_with_db_retries * mypy; fixed tests * fix task_adoption_timeout in celery integration test * addressing comments * remove useless print * fix typo * move failure logic to executor * fix scheduler job test * adjustments for new scheduler job * appeasing static checks * fix test for new scheduler job paradigm * Updating docs for deprecations * news & small changes * news & small changes * Update newsfragments/30375.significant.rst Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Update newsfragments/30375.significant.rst Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * added cleanup stuck task functionality to base executor * fix sloppy mistakes & mypy * removing self.fail from base_executor * Update airflow/jobs/scheduler_job_runner.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Update airflow/jobs/scheduler_job_runner.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Fix job_id filter * Don't even run query if executor doesn't support timing out queued tasks * Add support for LocalKubernetesExecutor and CeleryKubernetesExecutor * Add config option to control how often it runs - we want it quicker than the timeout * Fixup newsfragment * mark old KE pending pod check interval as deprecated by new check interval * Fixup deprecation warnings This more closely mirrors how deprecations are raised for "normal" deprecations. I've removed the depth, as moving up the stack doesn't really help the user at all in this situation. * Another deprecation cleanup * Remove db retries * Fix test --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Jed Cunningham <jedcunningham@apache.org> Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
2 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I accidentally closed #30108, so this is basically reopening that PR. Some updates:
scheduler_job.py.celery.stalled_task_timeout,kubernetes.worker_pods_pending_timeout, andcelery.task_adoption_timeoutcloses: #28120
closes: #21225
closes: #28943
Tasks occasionally get stuck in queued and aren't resolved by
stalled_task_timeout(#28120). This PR moves the logic for handling stalled tasks to the scheduler and simplifies the logic by marking any task that has been queued for more thanscheduler.task_queued_timeoutas failed, allowing it to be retried if the task has available retries.This doesn't require an additional scheduler nor allow for the possibility of tasks to get stuck in an infinite loop of scheduled -> queued -> scheduled ... -> queued as exists in #28943.