Skip to content

[Feature][PyDolphinScheduler] Support DVC task in pyds #11922#11941

Merged
caishunfeng merged 2 commits intoapache:devfrom
jieguangzhou:pyds-dvc
Sep 17, 2022
Merged

[Feature][PyDolphinScheduler] Support DVC task in pyds #11922#11941
caishunfeng merged 2 commits intoapache:devfrom
jieguangzhou:pyds-dvc

Conversation

@jieguangzhou
Copy link
Copy Markdown
Member

@jieguangzhou jieguangzhou commented Sep 14, 2022

Purpose of the pull request

close: #11922

Brief change log

  • add pyds DVC task
  • add YAML example
  • add doc
  • optimize DVC task plugin code

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

@jieguangzhou
Copy link
Copy Markdown
Member Author

jieguangzhou commented Sep 15, 2022

@zhongjiajie @caishunfeng PTAL, thanks

@SbloodyS SbloodyS added the feature new feature label Sep 16, 2022
@SbloodyS SbloodyS added this to the 3.1.0 milestone Sep 16, 2022
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Sep 16, 2022 •
edited
Loading

Codecov Report

Merging #11941 (d0676ae) into dev (e50a32d) will decrease coverage by 0.07%.
The diff coverage is 15.00%.

@@             Coverage Diff              @@
##                dev   #11941      +/-   ##
============================================
- Coverage     38.73%   38.65%   -0.08%     
+ Complexity     4019     4004      -15     
============================================
  Files          1003     1002       -1     
  Lines         37225    37213      -12     
  Branches       4237     4249      +12     
============================================
- Hits          14418    14385      -33     
- Misses        21186    21195       +9     
- Partials       1621     1633      +12
Impacted Files Coverage Δ
...dolphinscheduler/plugin/task/dvc/DvcConstants.java 0.00% <0.00%> (ø)
...olphinscheduler/plugin/task/dvc/DvcParameters.java 0.00% <0.00%> (-100.00%) ⬇️
...ache/dolphinscheduler/plugin/task/dvc/DvcTask.java 65.21% <75.00%> (ø)
...eduler/server/worker/task/WorkerHeartBeatTask.java 69.38% <0.00%> (-8.17%) ⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java 55.12% <0.00%> (-1.29%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Sep 16, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

68.5% 68.5% Coverage
0.0% 0.0% Duplication

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Sep 17, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

68.5% 68.5% Coverage
0.0% 0.0% Duplication

caishunfeng
caishunfeng approved these changes Sep 17, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

zhongjiajie
zhongjiajie approved these changes Sep 17, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@caishunfeng caishunfeng merged commit bbc228e into apache:dev Sep 17, 2022
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Sep 17, 2022
@jieguangzhou @xdu-chenrj
[Feature][PyDolphinScheduler] Support DVC task in pyds apache#11922 (a…
1fcfb24 
…pache#11941)

* add dvc task in pyds

* add BaseDVC class
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Sep 17, 2022
@jieguangzhou @xdu-chenrj
[Feature][PyDolphinScheduler] Support DVC task in pyds apache#11922 (a…
23dd244 
…pache#11941)

* add dvc task in pyds

* add BaseDVC class
zhongjiajie pushed a commit that referenced this pull request Sep 18, 2022
@jieguangzhou @zhongjiajie
[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)
dde6f63 
* add dvc task in pyds

* add BaseDVC class

(cherry picked from commit bbc228e)
@caishunfeng caishunfeng added the release cherry-pick Mark this issue/PR had cherry-pick for release version label Sep 19, 2022
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 13, 2022
@jieguangzhou @xdu-chenrj
[Feature][PyDolphinScheduler] Support DVC task in pyds apache#11922 (a…
6d15b36 
…pache#11941)

* add dvc task in pyds

* add BaseDVC class
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@caishunfeng caishunfeng caishunfeng approved these changes

@zhongjiajie zhongjiajie zhongjiajie approved these changes

@SbloodyS SbloodyS Awaiting requested review from SbloodyS SbloodyS is a code owner

@zhuangchong zhuangchong Awaiting requested review from zhuangchong

Assignees

@jieguangzhou jieguangzhou

Labels

backend document feature new feature Pyscheduler release cherry-pick Mark this issue/PR had cherry-pick for release version

Projects

None yet

Milestone

3.1.0

Development

Successfully merging this pull request may close these issues.

[Feature][PyDolphinScheduler] Support DVC task in pyds

5 participants

@jieguangzhou @codecov-commenter @caishunfeng @zhongjiajie @SbloodyS