Add a concurrency model with ThreadPoolExecutor #5011
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Thank you for the review, I will apply your recommandation in the next week. |
Contributor
Author
|
Not yet. I am doing it right now. |
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
I don't know if it would be useful to someone else, but I needed to implement a concurrency model based on thread (and not processes) in celery because I wanted to pass future objects between tasks and some coroutines and it is not pickable.
Of course it is not scale out and it has limitations (because of the GIL), but it can be useful for people who wants the share memory between tasks and another thread (for example the asyncio event loop) without blocking as the 'solo' concurrency model.
I am open to comment. Maybe this model won't be needed anymore in celery 5 as it could be replaced by an asyncio loop, which is not possible in celery 4.
And I would be happy to help on a massive asyncio refactoring for Celery 5.
Regards