Skip to content

Move async user tasks to thread#7339

Closed
fjetter wants to merge 3 commits intodask:mainfrom
fjetter:move_async_to_thread
Closed

Move async user tasks to thread#7339
fjetter wants to merge 3 commits intodask:mainfrom
fjetter:move_async_to_thread

Conversation

@fjetter
Copy link
Copy Markdown
Member

@fjetter fjetter commented Nov 21, 2022

This came up in #7320 in context of cancellation of these tasks during worker closing

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Nov 21, 2022

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       20 files  ±0         20 suites  ±0   11h 28m 28s ⏱️ - 30m 24s
  3 754 tests ±0    3 641 ✔️  -   2     106 💤 ±0    7 +  2 
36 314 runs  ±0  34 532 ✔️  - 29  1 748 💤 ±0  34 +29 

For more details on these failures, see this check.

Results for commit 2241969. ± Comparison against base commit 9255987.

♻️ This comment has been updated with latest results.

try:
ts.start_time = time()
if iscoroutinefunction(function):
result = await apply_function_async(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could now also remove the definition of (the now unused) apply_function_async

Copy link
Copy Markdown
Member

@hendrikmakait hendrikmakait left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks, @fjetter!

@gjoseph92
Copy link
Copy Markdown
Collaborator

+1 on this approach, I'd love to not let user code run in our event loop.

This also would have made #5997 a bit easier, since we'd lose the ability to cancel user async tasks (so we wouldn't have to special-case rescheduling them).

@fjetter
Copy link
Copy Markdown
Member Author

fjetter commented Dec 1, 2022

FYI this is currently soft blocked. Prefect is using this feature quite heavily and they observed some change in behavior relating to exception handling.
Since they are a big user of this feature and we're not in a rush, we'll wait a bit for them to look into it

@zanieb
Copy link
Copy Markdown
Contributor

zanieb commented Dec 6, 2022

@fjetter — just to confirm, this means that user-submitted async tasks are now running on multiple threads and event loops instead of a single thread and event loop?

@fjetter
Copy link
Copy Markdown
Member Author

fjetter commented Dec 8, 2022

just to confirm, this means that user-submitted async tasks are now running on multiple threads and event loops instead of a single thread and event loop?

Yes, exactly.

@zanieb
Copy link
Copy Markdown
Contributor

zanieb commented Jan 3, 2023

This should be good to go from our end! Thanks for letting us investigate.

@fjetter fjetter force-pushed the move_async_to_thread branch from f1028a1 to 2241969 Compare August 7, 2023 15:06
@fjetter fjetter closed this Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants