Skip to content

#203 add sqoop task#1974

Merged
Eights-Li merged 14 commits intoapache:devfrom
zixi0825:add_sqoop
Feb 24, 2020
Merged

#203 add sqoop task#1974
Eights-Li merged 14 commits intoapache:devfrom
zixi0825:add_sqoop

Conversation

@zixi0825
Copy link
Copy Markdown
Member

What is the purpose of the pull request

This pull request adds sqoop task.

Brief change log

  • Add sqoop source type : mysql,hive,hdfs.
  • Add sqoop target type : mysql,hive,hdfs.
  • Add sqoop task ui.

Verify this pull request

This change added tests and can be verified as follows:

  • Added SqoopTaskTest to verify the change.

@zixi0825 zixi0825 requested a review from Eights-Li February 18, 2020 13:14
@codecov-io
Copy link
Copy Markdown

codecov-io commented Feb 18, 2020

Codecov Report

Merging #1974 into dev will increase coverage by 1.12%.
The diff coverage is 72.96%.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #1974      +/-   ##
============================================
+ Coverage     30.84%   31.96%   +1.12%     
- Complexity     1382     1511     +129     
============================================
  Files           294      311      +17     
  Lines         15022    15402     +380     
  Branches       1810     1864      +54     
============================================
+ Hits           4633     4923     +290     
- Misses         9930     9969      +39     
- Partials        459      510      +51
Impacted Files Coverage Δ Complexity Δ
...lphinscheduler/server/worker/task/TaskManager.java 0% <0%> (ø) 0 <0> (ø) ⬇️
...pache/dolphinscheduler/common/enums/QueryType.java 0% <0%> (ø) 0 <0> (?)
...phinscheduler/server/worker/task/AbstractTask.java 11.94% <0%> (-0.37%) 2 <0> (ø)
...hinscheduler/common/utils/TaskParametersUtils.java 73.68% <0%> (-4.1%) 11 <0> (ø)
...common/task/sqoop/targets/TargetHiveParameter.java 100% <100%> (ø) 17 <17> (?)
...common/task/sqoop/targets/TargetHdfsParameter.java 100% <100%> (ø) 13 <13> (?)
...apache/dolphinscheduler/common/enums/TaskType.java 80.95% <100%> (+0.95%) 2 <0> (ø) ⬇️
...ommon/task/sqoop/sources/SourceMysqlParameter.java 100% <100%> (ø) 19 <19> (?)
...common/task/sqoop/sources/SourceHdfsParameter.java 100% <100%> (ø) 3 <3> (?)
...common/task/sqoop/sources/SourceHiveParameter.java 100% <100%> (ø) 9 <9> (?)
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34e0a66...08985e0. Read the comment docs.

@Eights-Li
Copy link
Copy Markdown
Contributor

1 add license
2 add the test class to pom

Copy link
Copy Markdown
Contributor

@Eights-Li Eights-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add more UT

/**
* sqoop task extends the shell task
*/
public class SqoopTask extends ShellTask {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sqoop tasks will be converted to mapreduce tasks, managed by yarn, SqoopTask class need extends AbstractYarnTask not ShellTask

@zixi0825 zixi0825 closed this Feb 19, 2020
@zixi0825 zixi0825 reopened this Feb 19, 2020
@zixi0825 zixi0825 closed this Feb 19, 2020
@Eights-Li Eights-Li reopened this Feb 19, 2020
@sonarqubecloud
Copy link
Copy Markdown

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 14 Code Smells

45.7% 45.7% Coverage
1.6% 1.6% Duplication

@Jave-Chen
Copy link
Copy Markdown
Contributor

please make sure code have been checked by ali style plugin.

import java.util.List;

/**
* @author simfo
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the author and time


// StringBuilder srcWhere = new StringBuilder();
// if(sourceMysqlParameter.getSrcConditionList().size()>0){
// List<Property> items = sourceMysqlParameter.getSrcConditionList();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this comment out code

@Eights-Li
Copy link
Copy Markdown
Contributor

@break60 please review front-end~

@break60
Copy link
Copy Markdown
Contributor

break60 commented Feb 21, 2020

+1

@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 9 Code Smells

77.1% 77.1% Coverage
1.5% 1.5% Duplication

@zixi0825 zixi0825 requested a review from Eights-Li February 21, 2020 08:50
Copy link
Copy Markdown
Contributor

@Eights-Li Eights-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants