Skip to content

New Dask 2022.10.1. release broke Airlfow's integration tests #9609

@potiuk

Description

@potiuk

The new Dask release broke (due to bokeh removal) tests for Airflow Dask Executor integration. Since we have almost no knowledge of Dask, we have to defer it to your team to fix it.

In the meantime we just limit Dask to < 2022.10.1 in Airlfow: apache/airflow#27383 .
I would love if someone from the Dask team or especially Dask users could take a look and fix the tests as appropriate.

Example failing build here: https://github.com/apache/airflow/actions/runs/3356434694/jobs/5561625862#step:9:19064

Fixing it is as easy as getting our breeze env runnnig and updating to latest version of dask there and runnint tests - easy steps explained in https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst - an easy way to become an Airflow contributor.

Also I think Dask team should seriously consider if they want to start maintaining it separately from Airflow.
We have virtually no experience of Dask in the maintainer team, and we already started discussions in the past to deprecate the executor regardless (and we might soon come back to the discussion) - I wonder if that is something you would like to take on that one. We have some examples of integrations with Airlfow (for example Great Expectations) where they maintain Airlfow integration in their own packages (precisely for the reason of maintaining future compatibility and keeping up with their deprecations/changes).

Describe the issue:

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:

Environment:

  • Dask version:
  • Python version:
  • Operating System:
  • Install method (conda, pip, source):

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageNeeds a response from a contributor

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions