Skip to content

drop deprecated tornado.netutil.ExecutorResolver#6031

Merged
fjetter merged 1 commit intodask:mainfrom
graingert:backport-default-loop-resolver
Apr 5, 2022
Merged

drop deprecated tornado.netutil.ExecutorResolver#6031
fjetter merged 1 commit intodask:mainfrom
graingert:backport-default-loop-resolver

Conversation

@graingert
Copy link
Member

ExecutorResolver was deprecated in tornado v5.0 and causes
a number of problems for us:

  • it results in a thread leak as the executor is never shutdown
    and so requires a .warmup() from the resource tracker
  • .warmup() instantiates the resolver and attaches an io_loop
    tornadoweb/tornado@43bd9ce
    this results in a DeprecationWarning on 3.10 because there's no
    loop running

this swaps it for a backport of DefaultLoopResolver from tornado v6.2
that uses get_running_loop().getaddrinfo for resolving DNS

  • Closes #xxxx
  • Tests added / passed
  • Passes pre-commit run --all-files

ExecutorResolver was deprecated in tornado v5.0 and causes
a number of problems for us:

- it results in a thread leak as the executor is never shutdown
and so requires a .warmup() from the resource tracker
- .warmup() instantiates the resolver and attaches an io_loop
  tornadoweb/tornado@43bd9ce
  this results in a DeprecationWarning on 3.10 because there's no
  loop running

this swaps it for a backport of DefaultLoopResolver from tornado v6.2
that uses get_running_loop().getaddrinfo for resolving DNS
@github-actions
Copy link
Contributor

Unit Test Results

       18 files  ±0         18 suites  ±0   9h 55m 36s ⏱️ +17s
  2 703 tests ±0    2 617 ✔️  - 4       82 💤 +1  4 +3 
24 163 runs  ±0  22 871 ✔️  - 3  1 287 💤 ±0  5 +3 

For more details on these failures, see this check.

Results for commit 454fd8f. ± Comparison against base commit 2ff681c.

@mrocklin
Copy link
Member

I don't know enough about this space to give a full review, but in general I'm happy to trust you on this one :)

@graingert graingert marked this pull request as ready for review March 31, 2022 14:31
@graingert graingert requested a review from fjetter April 5, 2022 09:40
Copy link
Member

@fjetter fjetter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised that we no longer need the warmup but if it works, I'm OK with it

Comment on lines 1614 to -1616
"""Context manager to ensure we haven't leaked any threads"""
# "TCP-Executor" threads are never stopped once they are started
BaseTCPConnector.warmup()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this not failing?

Copy link
Member Author

@graingert graingert Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the threads used by await asyncio.get_running_loop().getaddrinfo( are shutdown when the event loop is closed

@graingert graingert requested a review from fjetter April 5, 2022 13:40
@fjetter fjetter merged commit f4c52e9 into dask:main Apr 5, 2022
@graingert graingert deleted the backport-default-loop-resolver branch April 5, 2022 14:25
graingert added a commit to graingert/distributed that referenced this pull request Apr 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants