Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

By Integrating Apache Dolphin Scheduler with Apache Zeppelin through zeppelin task plugin, we aim to provide Dolphin users, especially those big data engineers, with so-called Big Data Studio experience, which means users can develop and debug big data related tasks in zeppelin notebook interactively and schedule them directly from dolphin with 'one click'. This feature will significantly boost development efficiency of big data engineers and lower the bar for those who do not have much experience in the big data area.

However, currently dolphin only has basic integration with zeppelin and we need more features in zeppelin task plugin to achieve our goal. Here are a few points I come up with at the moment:

• Enable note-level zeppelin task scheduling, which is already described here: [Feature][Task Plugin] Enable note-level task scheduling for zeppelin task #9798
• Add authentication feature to zeppelin task plugin through Configuration Center: [DSIP-19][Feature] Add connection center feature for DS #10283
• Add custom variables support in zeppelin task plugin. See zeppelin dynamic form for details. [Feature][Task Plugin] Add support for dynamic form for zeppelin task plugin #9977
• Enable users to switch zeppelin server endpoint. Specifically speaking, add an endpoint field in UI and by filling different endpoints, users use dolphin to schedule zeppelin tasks to zeppelin servers deployed on different remote clusters. In this way, once configurations are completed on zeppelin side, users will be able to schedule big data tasks on different environment without worrying about configurations. [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks #10925
• Increase stability in prod env. [Feature][Task Plugin] Zeppelin Task will run a duplicate note instead of the original one #10584

I would like to invite Apache Zeppelin PMC, Jeff Zhang @zjffdu to help with the review.

BTW, the idea of zeppelin task plugin is inspired by Jeff's previous work on Apache Airflow Zeppelin Operator, kudos to Jeff.

Use case

• Already described above

Related issues

releated: #9201 #9798 #5271

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct