Skip to content

Conversation

@namevic
Copy link

@namevic namevic commented Apr 2, 2019

Fixes #2700
Currently on receiving SIGTERM task is killed cause the SIGTERM not handled in the task

@auvipy auvipy added this to the 4.4.0 milestone Jun 10, 2019
@auvipy auvipy merged commit 4d10194 into celery:master Jun 10, 2019
@auvipy
Copy link
Member

auvipy commented Jun 10, 2019

sorry this cause unit test failure :)

@islammohamed
Copy link

Is just a lambda with the logging call would handle that SIGTERM correctly?!

@namevic
Copy link
Author

namevic commented Aug 2, 2019 via email

@islammohamed
Copy link

islammohamed commented Aug 2, 2019

No, the lambda is what happens when the sigterm is receive. https://docs.python.org/2/library/signal.html
On Fri, 2 Aug 2019 at 15:02 Islam Mohamed> wrote: Is just a lambda with the logging call would handle that SIGTERM correctly?! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5423?email_source=notifications&email_token=AEBNDBMMVL24BWZC725TEG3QCQO4ZA5CNFSM4HC4V3P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3NRMLQ#issuecomment-517674542>, or mute the thread https://github.com/notifications/unsubscribe-auth/AEBNDBIMKHVHYC6CEOGBWZLQCQO4ZANCNFSM4HC4V3PQ .

Understood, Maybe my comment wasn't clear. I'm wondering how the signal handler which is doing nothing but logging would help to handle the SIGTERM for tasks correctly?

@namevic
Copy link
Author

namevic commented Aug 3, 2019

No, the lambda is only printing the log for the user
signal.signal(signal.SIGTERM, handle_sigterm)
This is the code such handling the SIGTERM

@tjarjoura
Copy link

No, the lambda is only printing the log for the user
signal.signal(signal.SIGTERM, handle_sigterm)
This is the code such handling the SIGTERM

I think he's asking why you're not doing anything besides logging. I would expect the worker to exit after finishing the current task but it looks like it would just log this and continue running indefinitely.

@islammohamed
Copy link

islammohamed commented Aug 13, 2019

No, the lambda is only printing the log for the user
signal.signal(signal.SIGTERM, handle_sigterm)
This is the code such handling the SIGTERM

I think he's asking why you're not doing anything besides logging. I would expect the worker to exit after finishing the current task but it looks like it would just log this and continue running indefinitely.

Exactly, logging wouldn't be enough and considered as a solution for handling SIGTERM properly

@namevic
Copy link
Author

namevic commented Aug 13, 2019

Let's breathe...

  1. Worker currently supports warm shutdown on SIGTERM
  2. The issue was that when the worker received SIGTERM he sent this SIGTERM to current running tasks.
  3. When the task received SIGTERM no one handled it and the task stopped immediately.

Now after my fix 1 and 2 happens as it happens previously.
My fix handles the SIGTERM and not killing the running task and the worker is waiting till the task finished. After that worker goes to shutdown.

@islammohamed
Copy link

Let's breathe...

  1. Worker currently supports warm shutdown on SIGTERM
  2. The issue was that when the worker received SIGTERM he sent this SIGTERM to current running tasks.
  3. When the task received SIGTERM no one handled it and the task stopped immediately.

Now after my fix 1 and 2 happens as it happens previously.
My fix handles the SIGTERM and not killing the running task and the worker is waiting till the task finished. After that worker goes to shutdown.

Thank you so much for explaining it, Makes total sense 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGTERM does not do a warm shutdown.

4 participants