Skip to content

Sqoop task optimization#2943

Merged
wen-hemin merged 6 commits intoapache:devfrom
Eights-Li:dev-sqoop-optimization
Jun 11, 2020
Merged

Sqoop task optimization#2943
wen-hemin merged 6 commits intoapache:devfrom
Eights-Li:dev-sqoop-optimization

Conversation

@Eights-Li
Copy link
Copy Markdown
Contributor

What is the purpose of the pull request

#2917 #2763
sqoop task optimization

Brief change log

  • Add CUSTOM and TAMELATE job for sqoop task
  • support hadoop params like mr job name, etc
  • suooprt sqoop advanced params like --direct & --fetch-size, etc
  • Add sqoop task front-end selection box cascade display
  • fix bug : java map fields and hive map fields generate error

Verify this pull request

  • /server/worker/task/sqoop/SqoopTaskTest.java

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jun 10, 2020

Codecov Report

Merging #2943 into dev will increase coverage by 0.94%.
The diff coverage is 60.86%.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #2943      +/-   ##
============================================
+ Coverage     37.20%   38.14%   +0.94%     
- Complexity     2559     2621      +62     
============================================
  Files           435      436       +1     
  Lines         20020    20079      +59     
  Branches       2426     2443      +17     
============================================
+ Hits           7449     7660     +211     
+ Misses        11904    11711     -193     
- Partials        667      708      +41     
Impacted Files Coverage Δ Complexity Δ
...ommon/task/sqoop/targets/TargetMysqlParameter.java 100.00% <ø> (ø) 19.00 <0.00> (ø)
...ver/master/consumer/TaskPriorityQueueConsumer.java 51.82% <0.00%> (-7.00%) 16.00 <0.00> (ø)
.../sqoop/generator/sources/MysqlSourceGenerator.java 58.00% <42.85%> (+58.00%) 5.00 <0.00> (+5.00)
...inscheduler/common/task/sqoop/SqoopParameters.java 74.00% <58.06%> (-6.00%) 25.00 <12.00> (+10.00) ⬇️
...worker/task/sqoop/generator/SqoopJobGenerator.java 74.07% <72.72%> (+74.07%) 10.00 <2.00> (+10.00)
...he/dolphinscheduler/common/enums/SqoopJobType.java 88.88% <88.88%> (ø) 3.00 <3.00> (?)
...r/worker/task/sqoop/generator/CommonGenerator.java 83.87% <89.47%> (+83.87%) 6.00 <0.00> (+6.00)
.../org/apache/dolphinscheduler/api/enums/Status.java 100.00% <100.00%> (ø) 5.00 <0.00> (ø)
.../sqoop/generator/targets/MysqlTargetGenerator.java 75.75% <100.00%> (+75.75%) 4.00 <0.00> (+4.00)
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2749c7e...a420644. Read the comment docs.

@Eights-Li
Copy link
Copy Markdown
Contributor Author

@break60 please front-end. thx
@zixi0825 hi, this pr for sqoop task
@743294668 bro, Can this pr meet your requirements?

@break60
Copy link
Copy Markdown
Contributor

break60 commented Jun 10, 2020

front-end +1

Comment on lines +71 to +73
if(sourceGenerator == null || targetGenerator == null){
return null;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If return null, I see the follow-up code, the task will eventually be judged as successful.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, if null should throw a runtimeException

@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 1 Code Smell

68.4% 68.4% Coverage
0.0% 0.0% Duplication

warning The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more here

@Eights-Li Eights-Li requested a review from wen-hemin June 11, 2020 00:56
Copy link
Copy Markdown
Contributor

@wen-hemin wen-hemin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@wen-hemin wen-hemin merged commit a7fd0a5 into apache:dev Jun 11, 2020
@Eights-Li Eights-Li deleted the dev-sqoop-optimization branch July 3, 2020 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants