I have a pipeline where part of the configuration is specifying the number of chunks to produce for a dataset. Because of one of the tools I’m using, I have to use many chunks and keep the chunk size relatively small. This means that the complete pipeline will execute tens of thousands of jobs, and the DAG takes a very long time to compile.

As a feature, is there any way the process of building the DAG could be multi-threaded to go faster?