[Feature][Task-Plugin]Add zeppelin task-plugin to support Apache Zeppelin (#9201) by EricGao888 · Pull Request #9327 · apache/dolphinscheduler

EricGao888 · 2022-04-02T06:00:17Z

Purpose of the pull request

Add Apache Zeppelin task plugin to support scheduling zeppelin notebook tasks.
This PR closes: [Feature][Task-Plugin] Add Apache Zeppelin Task Plugin #9201

Brief change log

This PR focuses on the basic function of zeppelin task plugin, enabling Dolphinscheduler to execute and schedule paragraphs of Apache Zeppelin notebook.
Currently there is no cancelNote method in Apache Zeppelin Client Api, so zeppelin task execution and scheduling is in paragraph level in this PR.
I will add more features in the future PRs, like session management, job execution progress query, authentication, etc.

Verify this pull request

Verify by unit tests. Since there's no front-end implementation at this time, I haven't performed manual test yet.

codecov-commenter · 2022-04-02T06:13:36Z

Codecov Report

Merging #9327 (2b3f717) into dev (4a29c6a) will decrease coverage by 0.26%.
The diff coverage is 41.47%.

@@             Coverage Diff              @@
##                dev    #9327      +/-   ##
============================================
- Coverage     39.98%   39.72%   -0.27%     
- Complexity     4375     4401      +26     
============================================
  Files           822      830       +8     
  Lines         32924    33256     +332     
  Branches       3656     3681      +25     
============================================
+ Hits          13165    13210      +45     
- Misses        18532    18817     +285     
- Partials       1227     1229       +2

Impacted Files	Coverage Δ
...lugin/alert/wechat/WeChatAlertParamsConstants.java	`0.00% <ø> (ø)`
...uler/plugin/alert/wechat/WechatAppChatMessage.java	`0.00% <0.00%> (ø)`
...cheduler/plugin/alert/wechat/WechatAppMessage.java	`0.00% <0.00%> (ø)`
...inscheduler/api/controller/ExecutorController.java	`40.00% <ø> (ø)`
...cheduler/api/service/impl/ExecutorServiceImpl.java	`41.96% <ø> (ø)`
...r/api/service/impl/ProcessInstanceServiceImpl.java	`60.22% <ø> (-0.12%)`	⬇️
...er/api/service/impl/TaskDefinitionServiceImpl.java	`24.01% <ø> (+0.05%)`	⬆️
...che/dolphinscheduler/common/enums/AlertStatus.java	`0.00% <0.00%> (ø)`
...nscheduler/service/process/ProcessServiceImpl.java	`30.76% <ø> (ø)`
...olphinscheduler/plugin/task/api/TaskConstants.java	`0.00% <ø> (ø)`
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab03570...2b3f717. Read the comment docs.

...n/src/test/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinParametersTest.java

EricGao888 · 2022-04-03T09:08:35Z

Unit tests added. The PR fails license-check by script tools/dependencies/check-LICENSE.sh. Am I supposed to add org.apache.zeppelin:zeppelin-client:0.10.1 org.apache.zeppelin:zeppelin-common:0.10.1 into known-dependencies.txt? May I ask is there anything else I need to do in order to fix it?

SbloodyS · 2022-04-06T04:36:31Z

Unit tests added. The PR fails license-check by script tools/dependencies/check-LICENSE.sh. Am I supposed to add org.apache.zeppelin:zeppelin-client:0.10.1 org.apache.zeppelin:zeppelin-common:0.10.1 into known-dependencies.txt? May I ask is there anything else I need to do in order to fix it?

Yes. you should add it to known-dependencies.txt

...sk-zeppelin/src/main/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinTask.java

dolphinscheduler-common/src/main/resources/common.properties

zhongjiajie · 2022-04-07T03:36:58Z

Unit tests added. The PR fails license-check by script tools/dependencies/check-LICENSE.sh. Am I supposed to add org.apache.zeppelin:zeppelin-client:0.10.1 org.apache.zeppelin:zeppelin-common:0.10.1 into known-dependencies.txt? May I ask is there anything else I need to do in order to fix it?

BTW, you should also add each jar package license into dolphinscheduler-dist/release-docs/licenses directory

EricGao888 · 2022-04-07T08:30:03Z

Unit tests added. The PR fails license-check by script tools/dependencies/check-LICENSE.sh. Am I supposed to add org.apache.zeppelin:zeppelin-client:0.10.1 org.apache.zeppelin:zeppelin-common:0.10.1 into known-dependencies.txt? May I ask is there anything else I need to do in order to fix it?

BTW, you should also add each jar package license into dolphinscheduler-dist/release-docs/licenses directory

@zhongjiajie Licenses added in the latest commit. May I ask is there any task level configuration files instead of common.properties I can use for zeppelin task plugin config? As discussed here: #9327 (comment)

tools/dependencies/known-dependencies.txt

EricGao888 · 2022-04-11T06:56:35Z

I'm not familiar with front-end. I will focus on the backend features of zeppelin task plugin in the following PRs. Anything else do I need to do with this PR? Seems all comments have been resolved. @zhongjiajie

zhongjiajie · 2022-04-14T06:29:10Z

Do you have time to take a look at this PR after you finish your 3.0.0-alpha released? @zhuangchong

zhuangchong · 2022-04-14T11:35:09Z

Do you have time to take a look at this PR after you finish your 3.0.0-alpha released

OK

zhuangchong · 2022-04-14T11:41:05Z

...pelin/src/main/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinParameters.java

+     * @see <a href="https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html#API_RunJobFlow_Examples">API_RunJobFlow_Examples</a>
+     */


The documentation link address is wrong.

Thx for the reminder. I will fix it.

The documentation link address is wrong.

@zhuangchong Fixed in the latest commit. PTAL when you have time. Thx!

…zeppelin/pom.xml (apache#9201)

…ed license (apache#9201)

sonarqubecloud · 2022-04-16T10:41:04Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
4 Code Smells

50.0% Coverage
0.0% Duplication

zhuangchong

+1

zhongjiajie

LGTM, thanks

zhongjiajie reviewed Apr 2, 2022

View reviewed changes

...n/src/test/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinParametersTest.java Outdated Show resolved Hide resolved

EricGao888 force-pushed the Fix-9201 branch from 6510995 to c6e8a29 Compare April 3, 2022 06:45

SbloodyS reviewed Apr 6, 2022

View reviewed changes

...sk-zeppelin/src/main/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinTask.java Outdated Show resolved Hide resolved

EricGao888 force-pushed the Fix-9201 branch from c6e8a29 to 2e06c26 Compare April 6, 2022 07:34

zhuangchong requested changes Apr 6, 2022

View reviewed changes

...sk-zeppelin/src/main/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinTask.java Show resolved Hide resolved

dolphinscheduler-common/src/main/resources/common.properties Show resolved Hide resolved

zhongjiajie assigned EricGao888 Apr 7, 2022

zhongjiajie added miss:docs missing documents in PR miss:ui missing UI components labels Apr 7, 2022

EricGao888 requested a review from zhuangchong April 7, 2022 09:18

EricGao888 force-pushed the Fix-9201 branch from 04eb3a6 to 81afd93 Compare April 7, 2022 14:23

EricGao888 requested a review from zhongjiajie April 8, 2022 01:51

zhongjiajie reviewed Apr 8, 2022

View reviewed changes

tools/dependencies/known-dependencies.txt Show resolved Hide resolved

EricGao888 requested a review from zhongjiajie April 8, 2022 05:20

zhuangchong reviewed Apr 14, 2022

View reviewed changes

EricGao888 added 8 commits April 16, 2022 18:20

Add zeppelin task-plugin to support Apache Zeppelin (apache#9201)

320195e

Add unit tests for zeppelin task plugin (apache#9201)

016f5d3

Fix imports in zeppelin task plugin (apache#9201)

cdd0ab2

Add zeppelin related jar into known-dependencies (apache#9201)

4efd23c

Fix known-dependencies related to zeppelin task plugin (apache#9201)

6c6bcc9

Add licenses for libs related to zeppelin task plugin (apache#9201)

1e51ee9

Update version in dolphinscheduler-task-plugin/dolphinscheduler-task-…

564a49f

…zeppelin/pom.xml (apache#9201)

Update dolphinscheduler-dist/release-docs/LICENSE with zeppelin relat…

f0cfe30

…ed license (apache#9201)

Fix doc link address in ZeppelinTask.java (apache#9201)

a071263

EricGao888 force-pushed the Fix-9201 branch from 2b3f717 to a071263 Compare April 16, 2022 10:20

zhuangchong approved these changes Apr 17, 2022

View reviewed changes

zhongjiajie approved these changes Apr 17, 2022

View reviewed changes

zhongjiajie merged commit a5bbf78 into apache:dev Apr 17, 2022

EricGao888 mentioned this pull request Apr 26, 2022

[Feature][Task Plugin] Enable note-level task scheduling for zeppelin task #9798

Closed

3 tasks

EricGao888 added feature new feature backend and removed miss:docs missing documents in PR miss:ui missing UI components labels Aug 31, 2022

		* @see <a href="https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html#API_RunJobFlow_Examples">API_RunJobFlow_Examples</a>
		*/

Conversation

EricGao888 commented Apr 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

codecov-commenter commented Apr 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

EricGao888 commented Apr 3, 2022

Uh oh!

SbloodyS commented Apr 6, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhongjiajie commented Apr 7, 2022

Uh oh!

EricGao888 commented Apr 7, 2022

Uh oh!

Uh oh!

EricGao888 commented Apr 11, 2022

Uh oh!

zhongjiajie commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhuangchong commented Apr 14, 2022

Uh oh!

zhuangchong Apr 14, 2022

Choose a reason for hiding this comment

Uh oh!

EricGao888 Apr 15, 2022

Choose a reason for hiding this comment

Uh oh!

EricGao888 Apr 16, 2022

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 16, 2022

Uh oh!

zhuangchong left a comment

Choose a reason for hiding this comment

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

EricGao888 commented Apr 2, 2022 •

edited

Loading

codecov-commenter commented Apr 2, 2022 •

edited

Loading

zhongjiajie commented Apr 14, 2022 •

edited

Loading