Shutdown kombu consumer on worker shutdown #5338
Shutdown kombu consumer on worker shutdown #5338cognifloyd merged 9 commits intoStackStorm:masterfrom
Conversation
cognifloyd
left a comment
There was a problem hiding this comment.
Nice.
For other reviewers. Here are the docs for should_stop:
https://docs.celeryproject.org/projects/kombu/en/stable/reference/kombu.mixins.html#kombu.mixins.ConsumerMixin.should_stop
arm4b
left a comment
There was a problem hiding this comment.
Thanks @khushboobhatia01, appreciate the contribution based on running st2 in HA prod!
Could you please also add a CHANGELOG.rst record for this fix?
@armab Yes, updated. |
|
LGTM, but can you please also add a test case for it? |
@Kami Added test case. Can you please merge? Thank you! :) |
cognifloyd
left a comment
There was a problem hiding this comment.
An excellent test. It feels more like an integration test though. Could you move the new test to a new file under st2actions/tests/integration?
Thanks!
|
|
Done! |
As of now, only the disptacher is shutdown in the shutdown method. This change takes care of shutting down Kombu consumer thread.
In HA environment this will ensure other healthy action runner pods to pick up requested actions instead of an unhealthy pod picking the action from queue and abandoning it.
In a non-HA environment, actions requested after worker has stopped will be in scheduled state and will be picked after new action runner comes up.