Description

Add Python API for DolphinScheduler, which allow user define workflow using Python code, aka workflows-as-code. And this is parent list to connect all children issues, until we public first release to pypi

• mailing-list: https://lists.apache.org/thread.html/rb54aea6cde22d288fb3fefa4b921f155c8237b5bf5b42e4081ba789b%40%3Cdev.dolphinscheduler.apache.org%3E

Functional

• First implementation of workflow-as-codes, including create process definition, tasks, setting tasks relation, run process definition. Add Python API implementation of workflows-as-code #6269
• How to update workflow, including task name(would create new task when change task name), task other attribute such as command, relations Add Python API implementation of workflows-as-code #6269.
• Add Missing Parameter
  • [python] Add param workgroup to process definition #6667
  • [python] Add parameter schedule for process definition #6664
• Add Example How To Use,
  • Bluck Define, [python] Add example about how to bulk create workflow #6659
  • Tutorial, Add Python API implementation of workflows-as-code #6269
• [Feature][python] Gateway deployment in production mode #7539
• 🆘 🆘 🆘 🆘 🆘 help wanted here Support All Task Type 🆘 🆘 🆘 🆘 🆘
  • 🔥 sub_process [Feature][Python] Add workflow as code task type sub_process #6930
  • 🔥 procedure [Feature][Python] Add workflow as code task type procedure #6929
  • 🔥 sql [Feature][Python] Add workflow as code task type sql #6924
  • 🔥 spark [Feature][Python] Add workflow as code task type spark #7346
  • flink [Feature][Python] Add workflow as code task type flink #7347
  • MR [Feature][Python] Add workflow as code task type MR #7348
  • 🔥 python [python] Add task type python http #6906
  • 🔥 dependent [Feature][Python] Add workflow as code task type dependent #6926
  • 🔥 http [python] Add task type python http #6906
  • datax [Feature][Python] Add workflow as code task type datax #7349
  • 🔥 conditions [Feature][Python] Add workflow as code task type conditions #6927
  • 🔥 switch [Feature][Python] Add workflow as code task type switch #6928

Dev

• Code style, currently I think we should use black it better if bot could auto fix it.
  • [python] Add flake8 and black for code style and integrated to GA #6679
  • [ci][python] Add isort to sort Python API imports #6871
  • [ci][python] Add coverage check in CI #6861
• Unit Test.
  • Add CI test for python API #6636
  • Coverage up to 85%, add CI failed if coverage down to 85%, [ci][python] Add coverage check in CI #6861
• Add custom
  • exception [python] Add custom exception #6935

Doc

• Add docs to pyds, currently maybe use sphinx. [python] Initiate document for pydolphinscheduler #8005

What we have to consider carefully

1. A good mechanism to update process definition: For now DS design base on UI define workflow, which we modify database model step by step, is a quick different from define or update by code. We have to consider who to update it. Cause we should not ask user define process definition code or tasks code in the python API, I currently think maybe we should delete all tasks in process definition and then create new. It's still could not handle when users change project name and process definition, but we would try to fix it later
2. How to combine UI and codes when define and change workflow: We could update workflow define by UI or by edit Python API code after we create it, and how to combine them. If would cause consistency issues when we create workflow by python api but change them in UI, cause UI modify directly change database metadata but python api do not know and still in the old version. In python api define way, python file just like template, it parse the code and persist to database