Skip to content

[Feature][Task Plugin] Add mlflow task plugin for MLops scenario.#10036

Merged
caishunfeng merged 22 commits intoapache:devfrom
jieguangzhou:dev
May 19, 2022
Merged

[Feature][Task Plugin] Add mlflow task plugin for MLops scenario.#10036
caishunfeng merged 22 commits intoapache:devfrom
jieguangzhou:dev

Conversation

@jieguangzhou
Copy link
Copy Markdown
Member

@jieguangzhou jieguangzhou commented May 15, 2022

Purpose of the pull request

Add MLflow task plugin to enable Dolphin users to schedule MLflow.

  • support basic algorithm (lr, svm, lightgbm, xgboost)
  • support automl tool (autosklearn, flaml)

This PR relates to #9725.

Because I want to continue enriching the capabilities of this plugin, leave the issue open for now.

Brief change log

image

Verify this pull request

Verified by unit test.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 15, 2022

Codecov Report

Merging #10036 (caab81d) into dev (3e471f8) will increase coverage by 0.24%.
The diff coverage is 37.81%.

@@             Coverage Diff              @@
##                dev   #10036      +/-   ##
============================================
+ Coverage     40.87%   41.11%   +0.24%     
- Complexity     4633     4686      +53     
============================================
  Files           839      844       +5     
  Lines         34005    34342     +337     
  Branches       3769     3850      +81     
============================================
+ Hits          13898    14119     +221     
- Misses        18777    18862      +85     
- Partials       1330     1361      +31     
Impacted Files Coverage Δ
...nscheduler/plugin/task/mlflow/MlflowConstants.java 0.00% <0.00%> (ø)
...cheduler/plugin/task/mlflow/MlflowTaskChannel.java 0.00% <0.00%> (ø)
...r/plugin/task/mlflow/MlflowTaskChannelFactory.java 0.00% <0.00%> (ø)
...olphinscheduler/plugin/task/mlflow/MlflowTask.java 23.52% <23.52%> (ø)
...scheduler/plugin/task/mlflow/MlflowParameters.java 53.62% <53.62%> (ø)
...dolphinscheduler/remote/future/ResponseFuture.java 81.96% <0.00%> (-1.64%) ⬇️
.../org/apache/dolphinscheduler/api/enums/Status.java 100.00% <0.00%> (ø)
.../dolphinscheduler/plugin/task/flink/FlinkTask.java 67.93% <0.00%> (+2.47%) ⬆️
...erver/master/runner/task/TaskProcessorFactory.java 46.15% <0.00%> (+4.48%) ⬆️
...nscheduler/api/service/impl/TenantServiceImpl.java 79.87% <0.00%> (+5.34%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e471f8...caab81d. Read the comment docs.

@SbloodyS SbloodyS added backend UI ui and front end related labels May 15, 2022
@songjianet songjianet added the feature new feature label May 15, 2022
@labbomb
Copy link
Copy Markdown
Member

labbomb commented May 16, 2022

Please run the prettier directive to format the code

labbomb
labbomb previously approved these changes May 16, 2022
Copy link
Copy Markdown
Member

@labbomb labbomb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Member

@kezhenxu94 kezhenxu94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits

Tianqi-Dotes
Tianqi-Dotes previously approved these changes May 18, 2022
@jieguangzhou
Copy link
Copy Markdown
Member Author

jieguangzhou commented May 18, 2022

I have dealt with the problems above. PTAL @kezhenxu94 @zhongjiajie @caishunfeng , Thanks.

zhongjiajie
zhongjiajie previously approved these changes May 18, 2022
Copy link
Copy Markdown
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@songjianet songjianet added this to the 3.1.0-alpha milestone May 18, 2022
devosend
devosend previously approved these changes May 18, 2022
Copy link
Copy Markdown
Contributor

@devosend devosend left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhongjiajie zhongjiajie requested a review from labbomb May 18, 2022 09:18
songjianet
songjianet previously approved these changes May 18, 2022
Copy link
Copy Markdown
Member

@songjianet songjianet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# Conflicts:
#	docs/configs/docsdev.js
#	dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml
#	dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts
#	dolphinscheduler-ui/src/views/projects/task/components/node/types.ts
#	dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts
@jieguangzhou jieguangzhou requested a review from zhongjiajie May 18, 2022 11:23
@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 13 Code Smells

42.0% 42.0% Coverage
1.6% 1.6% Duplication

Copy link
Copy Markdown
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, it's a good feature, thanks for your contribution.

@jieguangzhou
Copy link
Copy Markdown
Member Author

LGTM

I have rebased the upstream dev, PTAL, thanks.

Copy link
Copy Markdown
Member

@labbomb labbomb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@caishunfeng caishunfeng merged commit 9ab3beb into apache:dev May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend feature new feature UI ui and front end related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Discussion][Machine Learning] Support AI task and the open source project about MLops

10 participants