[feature][task] Enable zeppelin schedule a whole zeppelin note by EricGao888 · Pull Request #10434 · apache/dolphinscheduler

EricGao888 · 2022-06-13T18:16:51Z

Purpose of the pull request

Currently, zeppelin task plugin only allows users to schedule one zeppelin paragraph at a time. After this PR, users will be able to schedule a whole zeppelin note, which is composed of multiple paragraphs at a time.
This PR closes: [Feature][Task Plugin] Enable note-level task scheduling for zeppelin task #9798

Brief change log

Already described above.

Verify this pull request

Verified by both unit test and manual test.

codecov-commenter · 2022-06-13T18:39:11Z

Codecov Report

Merging #10434 (c2d7eb5) into dev (190f253) will increase coverage by 0.03%.
The diff coverage is 36.36%.

@@             Coverage Diff              @@
##                dev   #10434      +/-   ##
============================================
+ Coverage     40.55%   40.58%   +0.03%     
- Complexity     4774     4776       +2     
============================================
  Files           878      878              
  Lines         35741    35769      +28     
  Branches       3969     3974       +5     
============================================
+ Hits          14493    14516      +23     
+ Misses        19805    19803       -2     
- Partials       1443     1450       +7

Impacted Files	Coverage Δ
...inscheduler/plugin/task/zeppelin/ZeppelinTask.java	`48.68% <34.37%> (-7.92%)`	⬇️
...duler/plugin/task/zeppelin/ZeppelinParameters.java	`84.61% <100.00%> (+7.69%)`	⬆️
...er/master/dispatch/host/assign/RandomSelector.java	`77.77% <0.00%> (-5.56%)`	⬇️
...er/api/service/impl/TaskDefinitionServiceImpl.java	`24.01% <0.00%> (ø)`
...che/dolphinscheduler/api/python/PythonGateway.java	`17.00% <0.00%> (+7.76%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 190f253...c2d7eb5. Read the comment docs.

EricGao888 · 2022-06-14T01:44:11Z

I'm not sure why E2E fails for resource center test. Could anyone plz help restart the test? Thx

SbloodyS · 2022-06-14T01:46:05Z

There are some flaky in FileManageE2ETest. I've restarted the CI. @EricGao888

caishunfeng · 2022-06-14T02:47:27Z

...sk-zeppelin/src/main/java/org/apache/dolphinscheduler/plugin/task/zeppelin/ZeppelinTask.java

+                                    paragraphResult.getParagraphId(),
+                                    paragraphResult.getResultInText()));
+                    status = paragraphResult.getStatus();
+                    if (status != Status.FINISHED) {


Does it mean failure if status is not FINISHED? If so, I think it's better to add some comments.

Sure, comments added in the latest commit.

dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-zeppelin.ts

EricGao888 · 2022-06-16T09:57:56Z

@SbloodyS Could you please take a look when you have time? Thx~

SbloodyS

LGTM overall. But we still have unresolved comment in #10434 (comment)

EricGao888 · 2022-06-16T10:01:22Z

LGTM overall. But we still have unresolved comment in #10434 (comment)

I've already removed all the redundant commented code in the latest commit.

EricGao888 · 2022-06-16T10:02:48Z

LGTM overall. But we still have unresolved comment in #10434 (comment)

Ops, I think I missed one comment. Will add some changes in the next commit.

EricGao888 · 2022-06-16T10:57:52Z

Looks like some kind of timezone related issue?

… task (apache#9798)

…re (apache#9798)

…sk UI (apache#9798)

…ecution status (apache#9798)

sonarqubecloud · 2022-06-17T03:42:36Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

42.9% Coverage
0.0% Duplication

zhongjiajie · 2022-06-18T04:45:39Z

There are some flaky in FileManageE2ETest. I've restarted the CI. @EricGao888

@SbloodyS what is that? and could we find some way to fix it?

zhongjiajie · 2022-06-18T04:48:50Z

Looks like some kind of timezone related issue?

@EricGao888 seem we already fix in #10284, could you please rebase on upstream dev and try again?

EricGao888 · 2022-06-18T04:49:55Z

Looks like some kind of timezone related issue?

@EricGao888 seem we already fix in #10284, could you please rebase on upstream dev and try again?

@zhongjiajie I had already rebased and the CI succeeded. Thx for the reminder.

zhongjiajie · 2022-06-18T04:57:09Z

LGTM overall. But we still have unresolved comment in #10434 (comment)

It already added and seem this PR ready to go now is it? @SbloodyS @EricGao888

EricGao888 · 2022-06-18T04:59:31Z

@zhongjiajie I think so, comments already added in the latest PR

SbloodyS · 2022-06-18T05:04:49Z

There are some flaky in FileManageE2ETest. I've restarted the CI. @EricGao888

@SbloodyS what is that? and could we find some way to fix it?

It have been fixed in #10450.

SbloodyS · 2022-06-18T05:05:28Z

LGTM overall. But we still have unresolved comment in #10434 (comment)

It already added and seem this PR ready to go now is it? @SbloodyS @EricGao888

Yes.

zhongjiajie

LGTM, Thanks

…e#10434)

EricGao888 requested review from Amy0104, SbloodyS, Tianqi-Dotes, caishunfeng, songjianet, zhongjiajie and zhuangchong as code owners June 13, 2022 18:16

caishunfeng reviewed Jun 14, 2022

View reviewed changes

EricGao888 force-pushed the Fix-9798 branch 2 times, most recently from 9cee096 to 78f2539 Compare June 14, 2022 12:59

EricGao888 requested a review from caishunfeng June 15, 2022 01:31

SbloodyS previously approved these changes Jun 16, 2022

View reviewed changes

EricGao888 dismissed SbloodyS’s stale review via e773bf1 June 16, 2022 10:12

EricGao888 added 4 commits June 17, 2022 11:13

[Feature][Task Plugin] Enable note-level task scheduling for zeppelin…

5dd12f9

… task (apache#9798)

[Feature][Task Plugin] Update docs for zeppelin note scheduling featu…

e6c2f32

…re (apache#9798)

[Feature][Task Plugin] Remove commented code in zeppelin / jupyter ta…

e40e19d

…sk UI (apache#9798)

[Feature][Task Plugin] Add comment in ZeppelinTask.java about note ex…

4f53e78

…ecution status (apache#9798)

EricGao888 force-pushed the Fix-9798 branch from e773bf1 to 4f53e78 Compare June 17, 2022 03:21

zhongjiajie added this to the 3.1.0-alpha milestone Jun 18, 2022

zhongjiajie approved these changes Jun 18, 2022

View reviewed changes

zhongjiajie merged commit 4be3b87 into apache:dev Jun 18, 2022

zhongjiajie changed the title ~~[Feature][Task Plugin] Enable zeppelin task to schedule a whole zeppelin note~~ [feature][task] Enable zeppelin schedule a whole zeppelin note Jun 18, 2022

hstdream pushed a commit to hstdream/dolphinscheduler that referenced this pull request Jun 23, 2022

[feature][task] Enable zeppelin schedule a whole zeppelin note (apach…

a7f431f

…e#10434)

Conversation

EricGao888 commented Jun 13, 2022

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

codecov-commenter commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

EricGao888 commented Jun 14, 2022

Uh oh!

SbloodyS commented Jun 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

caishunfeng Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

EricGao888 Jun 16, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EricGao888 commented Jun 16, 2022

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

EricGao888 commented Jun 16, 2022

Uh oh!

EricGao888 commented Jun 16, 2022

Uh oh!

EricGao888 commented Jun 16, 2022

Uh oh!

sonarqubecloud bot commented Jun 17, 2022

Uh oh!

zhongjiajie commented Jun 18, 2022

Uh oh!

zhongjiajie commented Jun 18, 2022

Uh oh!

EricGao888 commented Jun 18, 2022

Uh oh!

zhongjiajie commented Jun 18, 2022

Uh oh!

EricGao888 commented Jun 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SbloodyS commented Jun 18, 2022

Uh oh!

SbloodyS commented Jun 18, 2022

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jun 13, 2022 •

edited

Loading

SbloodyS commented Jun 14, 2022 •

edited

Loading

EricGao888 commented Jun 18, 2022 •

edited

Loading