Skip to content

Extensions vs Plugins #6011

@mrocklin

Description

@mrocklin

Currently we have two mechanisms to extend the scheduler and workers:

  1. Extensions: primarily used by Dask developers to modularize the scheduler and worker systems.
  2. Plugins: primarily used by Dask users to add new features to Dask from the user-side

Plugins contract

Plugins have an explicit contract and methods like transition, remove_worker, and so on. They are well defined.

Extension conventions

Extensions don't have such a contract but often do a few things by convention:

  1. Define some new handlers and inject those into the server's handlers
  2. Add themselves to server.extensions (although this is moved to a contract in Allow worker extensions to piggy-back on heartbeat #5957)

Well, that's pretty much it actually. In #5957 it's proposed to add a heartbeat mechanism .

Nothing here is set or required. Extensions are just classes that hang around. We don't really have any rules around them.

Open questions

  1. Should we try to merge these two so that there is only one extension mechanism?
  2. If not, should we try to formalize Extensions a little more?

This is low priority (I think). I'm raising this mostly to capture a conversation that was in a PR but didn't get resolved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions