Skip to content

if there's an exception in the Client async context manager body then close fast#6920

Merged
graingert merged 5 commits intodask:mainfrom
graingert:fast-close-in-client-aexit-on-error
Nov 2, 2022
Merged

if there's an exception in the Client async context manager body then close fast#6920
graingert merged 5 commits intodask:mainfrom
graingert:fast-close-in-client-aexit-on-error

Conversation

@graingert
Copy link
Copy Markdown
Member

@graingert graingert commented Aug 19, 2022

Closes #xxxx

  • Tests added / passed
  • Passes pre-commit run --all-files

@graingert graingert mentioned this pull request Aug 19, 2022
2 tasks
@graingert
Copy link
Copy Markdown
Member Author

graingert commented Aug 19, 2022

if anyone needs the old behaviour they can use a shield_from_exception context manager to transport the exception around the context manager:

import sys
import contextlib
from distributed import Client
import asyncio


@contextlib.asynccontextmanager
async def shield_from_exception(cmgr_factory, /, *args, **kwargs):
    err = None
    async with cmgr_factory(*args, **kwargs) as v:
        try:
            yield v
        except BaseException as e:
            err = e

    if err is not None:
        raise err


async def amain():
    async with shield_from_exception(Client, asynchronous=True) as client:
        raise Exception

def main():
    return asyncio.run(amain())

if __name__ == "__main__":
    sys.exit(main())


assert _close_proxy.mock_calls == [mock.call(fast=True)]
assert c.status == "closed"
assert (stop - start) < 2
Copy link
Copy Markdown
Member Author

@graingert graingert Aug 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find any tests that tested what _close(fast=True) does:

it just reduces this 2 second timeout to 0 (I think fast=True should be quite a bit more aggressive, eg force close rpc etc)

await asyncio.wait_for(handle_report_task, 0 if fast else 2)

so that's why I used a spy here

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would you mind incorporating a description of what closing fast means into the docstring for _close? I was wondering the same thing when I started reading this PR.

Copy link
Copy Markdown
Member Author

@graingert graingert Aug 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

introduced here https://github.com/dask/distributed/pull/27/files#diff-ce261e4d4b0185c8defbfcd46ebe8311a3ecb7ab81d653a2c6664c840340ab2cR250-R251 where it was used as a flag to leak tasks - however we now don't leak tasks but instead cancel them and wait for them

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hendrikmakait thanks for the review, it helped clarify my thinking on this!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would it make sense to rename fast to force?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think "forcefully=True" is probably the best here

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 19, 2022

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  +       15         15 suites  +15   6h 18m 10s ⏱️ + 6h 18m 10s
  3 072 tests +  3 072    2 987 ✔️ +  2 987    84 💤 +  84  1 +1 
22 737 runs  +22 737  21 750 ✔️ +21 750  986 💤 +986  1 +1 

For more details on these failures, see this check.

Results for commit fbf31d2. ± Comparison against base commit f07f384.

♻️ This comment has been updated with latest results.

Copy link
Copy Markdown
Member

@hendrikmakait hendrikmakait left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good to me, could you elaborate on the motivation for this change?


assert _close_proxy.mock_calls == [mock.call(fast=True)]
assert c.status == "closed"
assert (stop - start) < 2
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would you mind incorporating a description of what closing fast means into the docstring for _close? I was wondering the same thing when I started reading this PR.

@graingert graingert force-pushed the fast-close-in-client-aexit-on-error branch from c373746 to d6f6561 Compare August 23, 2022 10:34
Copy link
Copy Markdown
Member

@hendrikmakait hendrikmakait left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding the explanation. Feel free to ignore my nit, this just crossed my mind after reading your clarification.


assert _close_proxy.mock_calls == [mock.call(fast=True)]
assert c.status == "closed"
assert (stop - start) < 2
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would it make sense to rename fast to force?

@graingert graingert merged commit c137ac0 into dask:main Nov 2, 2022
@graingert graingert deleted the fast-close-in-client-aexit-on-error branch November 2, 2022 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants