-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
The new Dask release broke (due to bokeh removal) tests for Airflow Dask Executor integration. Since we have almost no knowledge of Dask, we have to defer it to your team to fix it.
In the meantime we just limit Dask to < 2022.10.1 in Airlfow: apache/airflow#27383 .
I would love if someone from the Dask team or especially Dask users could take a look and fix the tests as appropriate.
Example failing build here: https://github.com/apache/airflow/actions/runs/3356434694/jobs/5561625862#step:9:19064
Fixing it is as easy as getting our breeze env runnnig and updating to latest version of dask there and runnint tests - easy steps explained in https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst - an easy way to become an Airflow contributor.
Also I think Dask team should seriously consider if they want to start maintaining it separately from Airflow.
We have virtually no experience of Dask in the maintainer team, and we already started discussions in the past to deprecate the executor regardless (and we might soon come back to the discussion) - I wonder if that is something you would like to take on that one. We have some examples of integrations with Airlfow (for example Great Expectations) where they maintain Airlfow integration in their own packages (precisely for the reason of maintaining future compatibility and keeping up with their deprecations/changes).
Describe the issue:
Minimal Complete Verifiable Example:
# Put your MCVE code hereAnything else we need to know?:
Environment:
- Dask version:
- Python version:
- Operating System:
- Install method (conda, pip, source):