Skip to content

Conversation

@gpauloski
Copy link
Contributor

Description

This PR introduces plugins for Dask.distributed. Specifically, there is now a custom Dask.distributed Client which will automatically proxy large function arguments and results passed to or from Client.submit() and Client.map().

A new guide has been added discussion using ProxyStore with Dask.

Fixes N/A

Type of Change

  • Breaking Change (fix or enhancement which changes existing semantics of the public interface)
  • Enhancement (new features or improvements to existing functionality)
  • Bug (fixes for a bug or issue)
  • Internal (refactoring, style changes, testing, optimizations)
  • Documentation update (changes to documentation or examples)
  • Package (dependencies, versions, package metadata)
  • Development (CI workflows, pre-commit, linters, templates)
  • Security (security related changes)

Testing

Added new tests.

Pull Request Checklist

Please confirm the PR meets the following requirements.

  • Tags added to PR (e.g., breaking, bug, enhancement, internal, documentation, package, development, security).
  • Code changes pass pre-commit (e.g., mypy, ruff, etc.).
  • Tests have been added to show the fix is effective or that the new feature works.
  • New and existing unit tests pass locally with the changes.
  • Docs have been updated and reviewed if relevant.

@gpauloski gpauloski added the enhancement New feature or request label Dec 15, 2023
@gpauloski gpauloski force-pushed the custom-dask-client branch 2 times, most recently from 419b628 to c335d11 Compare December 15, 2023 00:09
Dask cannot serialize a Proxy type, so instead we pass factories as task
inputs and results and create proxies from those factories as needed.

Note that I tried a lot of methods for addressing Dask not serializing
Proxy types including overwriting/patching the Dask functions and
registering custom serializers. I could not get any of these to work so
I switched to the pass by factory method.
@gpauloski gpauloski merged commit a638674 into main Dec 15, 2023
@gpauloski gpauloski deleted the custom-dask-client branch December 15, 2023 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants