Skip to content

[Sqoop-Update]modify sqoop task param when upgrade ds to 1.3.4#4212

Merged
xingchun-chen merged 17 commits intoapache:1.3.4-preparefrom
Eights-Li:1.3.4-prepare-sqoop
Dec 12, 2020
Merged

[Sqoop-Update]modify sqoop task param when upgrade ds to 1.3.4#4212
xingchun-chen merged 17 commits intoapache:1.3.4-preparefrom
Eights-Li:1.3.4-prepare-sqoop

Conversation

@Eights-Li
Copy link
Copy Markdown
Contributor

@Eights-Li Eights-Li commented Dec 11, 2020

What is the purpose of the pull request

1 add --target-dir param to deal with import mysql to hive using sql query failed
2 modify sqoop task param when ds < 1.3.4 upgrade to 1.3.4
3 fix sqoop task import mysql to hive the web no hive selector

Brief change log

  • add --target-dir param to deal with import mysql to hive using sql query failed
  • modify sqoop task param when ds < 1.3.4 upgrade to 1.3.4
  • fix sqoop task import mysql to hive the web no hive selector

Verify this pull request

This change added tests and can be verified as follows:

  • worker/task/SqoopTaskTest

Eights-Li and others added 17 commits December 6, 2020 12:27
@Eights-Li
sqoop task optimization
04dc59b
@Eights-Li
sqoop front-end optimization
8834d8c
@Eights-Li
modify sqoop task UT
5d0982d
@Eights-Li
add sqoop task UT to pom
d3693cd
@Eights-Li
sqoop task source type or target type is null throw exception
93fb80a
@CalvinKirs @Eights-Li
fix testSqoopTaskTest bug (#3024)
bf23fa5
@Eights-Li
[FIX-4034][server] fix sqoop import fail (#4036)
27a22cd 
* fix #4043, sqoop import query fail

* fix #4043, sqoop task hard code & code style

* add license for SqoopConstants

* add private constructor for SqoopConstants

* fixed sqoop mysql pwd have special character

* fix checkstyle

* fix sqoop task log

* remove unused constants
@Eights-Li
[FIX-4034][server] fix sqoop import fail (#4036)
c80e200 
* fix #4043, sqoop import query fail

* fix #4043, sqoop task hard code & code style

* add license for SqoopConstants

* add private constructor for SqoopConstants

* fixed sqoop mysql pwd have special character

* fix checkstyle

* fix sqoop task log

* remove unused constants
@Eights-Li
fix sqoop task jdbc string contains special char (#4105)
c51a423
@Eights-Li
split sqoop import hive database and table (#4141)
da58048
@Eights-Li
modify JSONUtils
e3eeacc
@Eights-Li
cancel sqoop sql query import hive limit
bd8a8bd
@Eights-Li
update sqoop task params when upgrade to 1.3.4
2366097
@Eights-Li
sqoop hive target add target dir field
0420fe5
@Eights-Li
add target-dir param in sqoop task hive TargetHiveParameter
6609b91
@Eights-Li
Merge branch '1.3.4-prepare' of https://github.com/apache/incubator-d…
343b6f1 
…olphinscheduler into 1.3.4-prepare
@Eights-Li
fix hadoop params build failed in sqoop task
5a6e1d3
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Dec 11, 2020

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 5 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@lgcareer
Copy link
Copy Markdown
Contributor

lgcareer commented Dec 12, 2020

@Eights-Li I saw the ddl or dml under 1.3.4 is empty,so if 1.3.4 vresion didn't changed table schema or change table field values,so it's better not to create the 1.3.4_schema folder under sql/upgreade.

@xingchun-chen xingchun-chen added this to the 1.3.4-release milestone Dec 12, 2020
@Eights-Li
Copy link
Copy Markdown
Contributor Author

Eights-Li commented Dec 12, 2020

@Eights-Li I saw the ddl or dml under 1.3.4 is empty,so if 1.3.4 vresion didn't changed table schema or change table field values,so it's better not to create the 1.3.4_schema folder under sql/upgreade.

In ds-1.3.4, sqoop task meta data need to upgrade, when running upgrade code, need the schemaversion = 1.3.4, so I add the 1.3.4-schema dir in sql/update

@lgcareer
Copy link
Copy Markdown
Contributor

lgcareer commented Dec 12, 2020

@Eights-Li I saw the ddl or dml under 1.3.4 is empty,so if 1.3.4 vresion didn't changed table schema or change table field values,so it's better not to create the 1.3.4_schema folder under sql/upgreade.

In ds-1.3.4, sqoop task meta data need to upgrade, when running upgrade code, need the schemaversion = 1.3.4, so I add the 1.3.4-schema dir in sql/update

OK,Got it.

lgcareer
lgcareer approved these changes Dec 12, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@lgcareer lgcareer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@xingchun-chen
Copy link
Copy Markdown
Contributor

xingchun-chen commented Dec 12, 2020

+1

@xingchun-chen xingchun-chen merged commit 933eac7 into apache:1.3.4-prepare Dec 12, 2020
@Eights-Li Eights-Li deleted the 1.3.4-prepare-sqoop branch March 10, 2021 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lgcareer lgcareer lgcareer approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.3.4-release

Development

Successfully merging this pull request may close these issues.

4 participants

@Eights-Li @lgcareer @xingchun-chen @CalvinKirs