[feature] data synchronization function , through datax by wen-hemin · Pull Request #1881 · apache/dolphinscheduler

wen-hemin · 2020-02-02T04:48:06Z

please see this issue #1856

codecov-io · 2020-02-02T04:58:01Z

Codecov Report

❗ No coverage uploaded for pull request base (dev@ed5bc8c). Click here to learn what that means.
The diff coverage is 32%.

@@          Coverage Diff           @@
##             dev    #1881   +/-   ##
======================================
  Coverage       ?   24.11%           
======================================
  Files          ?      299           
  Lines          ?    14422           
  Branches       ?     2372           
======================================
  Hits           ?     3478           
  Misses         ?    10552           
  Partials       ?      392

Impacted Files	Coverage Δ
.../dolphinscheduler/api/service/ExecutorService.java	`31.87% <ø> (ø)`
...lphinscheduler/server/worker/task/TaskManager.java	`0% <ø> (ø)`
...heduler/server/master/runner/MasterExecThread.java	`13.3% <ø> (ø)`
...inscheduler/common/task/datax/DataxParameters.java	`0% <0%> (ø)`
...he/dolphinscheduler/common/model/DateInterval.java	`12.5% <0%> (ø)`
...che/dolphinscheduler/api/utils/ZooKeeperState.java	`0% <0%> (ø)`
...ler/server/master/runner/MasterTaskExecThread.java	`0% <0%> (ø)`
...dolphinscheduler/common/queue/TaskQueueZkImpl.java	`0% <0%> (ø)`
...lphinscheduler/server/worker/task/sql/SqlTask.java	`0% <0%> (ø)`
...olphinscheduler/common/model/TaskNodeRelation.java	`0% <0%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed5bc8c...ec234ee. Read the comment docs.

davidzollo · 2020-02-02T06:43:09Z

pstmt and resultset need close

https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=1881&resolved=false&types=BUG

qiaozhanwei

1，datax or sqoop is data import or export, what's mean ETL task naming ?
2，suggest remove class author，because other not

wen-hemin · 2020-02-02T07:27:29Z

1，datax or sqoop is data import or export, what's mean ETL task naming ?
2，suggest remove class author，because other not

1，The front end is named ETL, and the inside is implemented by Datax. It can be replaced by other implementations.
2，fixed

wen-hemin · 2020-02-02T07:28:10Z

pstmt and resultset need close

https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=1881&resolved=false&types=BUG

fixed

Eights-Li · 2020-02-02T10:27:35Z

...cheduler-common/src/main/java/org/apache/dolphinscheduler/common/task/etl/EtlParameters.java

+    /**
+     * datasource id
+     */
+    private int datasource;


datasource -> dataSource

thx, let me fix it

Eights-Li · 2020-02-02T10:28:19Z

...cheduler-common/src/main/java/org/apache/dolphinscheduler/common/task/etl/EtlParameters.java

+    /**
+     * datatarget id
+     */
+    private int datatarget;


datatarget -> dataTarget

thx, let me fix it

Eights-Li · 2020-02-02T10:36:56Z

...cheduler-common/src/main/java/org/apache/dolphinscheduler/common/task/etl/EtlParameters.java

+
+    @Override
+    public boolean checkParameters() {
+        if (!(datasource != 0 && StringUtils.isNotEmpty(dsType) && StringUtils.isNotEmpty(sql))) {


Whether these two if condition are considered to be combined into one？

thx, let me fix it

Eights-Li · 2020-02-02T10:58:45Z

1，datax or sqoop is data import or export, what's mean ETL task naming ?
2，suggest remove class author，because other not

1，The front end is named ETL, and the inside is implemented by Datax. It can be replaced by other implementations.
2，fixed

data sync as this task name is more suitable than etl?
Datax or sqoop should only be responsible for the import and export of data. Although it can take part in data process, the main data process should be implemented by hive-ql or spark.

Jave-Chen · 2020-02-03T04:47:19Z

New files UT coverage need add files at pom.xml at root. Like this:

 <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
        <includes>
            <include>**/common/utils/*.java</include>
            <include>**/common/threadutils/*.java</include>
            <include>**/common/graph/*.java</include>
            <include>**/common/queue/*.java</include>
            <include>**/api/utils/CheckUtilsTest.java</include>
            <include>**/api/utils/FileUtilsTest.java</include>
            <include>**/api/utils/exportprocess/DataSourceParamTest.java</include>
            <include>**/api/utils/exportprocess/DependentParamTest.java</include>
            <include>**/api/enums/*.java</include>
            <include>**/api/service/AccessTokenServiceTest.java</include>
            <include>**/api/service/QueueServiceTest.java</include>
            <include>**/api/service/MonitorServiceTest.java</include>
            <include>**/api/service/SessionServiceTest.java</include>
            <include>**/api/service/UsersServiceTest.java</include>
            <include>**/api/service/TenantServiceTest.java</include>
            <include>**/api/service/WorkerGroupServiceTest.java</include>
            <include>**/api/service/AlertGroupServiceTest.java</include>
            <include>**/api/service/ProjectServiceTest.java</include>
            <include>**/api/service/ProcessDefinitionServiceTest.java</include>
            <include>**/api/service/UdfFuncServiceTest.java</include>
            <include>**/api/service/ResourcesServiceTest.java</include>
            <include>**/api/service/ExecutorService2Test.java</include>
            <include>**/api/service/BaseServiceTest.java</include>
            <include>**/api/service/BaseDAGServiceTest.java</include>
            <include>**/alert/utils/ExcelUtilsTest.java</include>
            <include>**/alert/utils/FuncUtilsTest.java</include>
            <include>**/alert/utils/JSONUtilsTest.java</include>
            <include>**/alert/utils/PropertyUtilsTest.java</include>
            <include>**/server/utils/SparkArgsUtilsTest.java</include>
            <include>**/server/utils/FlinkArgsUtilsTest.java</include>
            <include>**/server/utils/ParamUtilsTest.java</include>
            <include>**/server/utils/ScheduleUtilsTest.java</include>
            <include>**/server/master/MasterExecThreadTest.java</include>
            <include>**/dao/mapper/AccessTokenMapperTest.java</include>
            <include>**/dao/mapper/AlertGroupMapperTest.java</include>
            <include>**/dao/mapper/AlertMapperTest.java</include>
            <include>**/dao/mapper/CommandMapperTest.java</include>
            <include>**/alert/template/AlertTemplateFactoryTest.java</include>
            <include>**/alert/template/impl/DefaultHTMLTemplateTest.java</include>
            <!-- add here -->
            <include>**/server/worker/task/AbstractTask.java</include>
        </includes>
        <!-- <skip>true</skip> -->
    </configuration>
</plugin>

# Conflicts: # pom.xml

wen-hemin · 2020-02-03T11:26:19Z

1，datax or sqoop is data import or export, what's mean ETL task naming ?
2，suggest remove class author，because other not

1，The front end is named ETL, and the inside is implemented by Datax. It can be replaced by other implementations.
2，fixed

data sync as this task name is more suitable than etl?
Datax or sqoop should only be responsible for the import and export of data. Although it can take part in data process, the main data process should be implemented by hive-ql or spark.

I think，A better name is DataX

Eights-Li · 2020-02-04T09:34:14Z

...nscheduler-server/src/test/java/org/apache/dolphinscheduler/server/utils/DataxUtilsTest.java

-        String targetColumn = "`select`";
-
-        assertEquals(DataxUtils.doConvertKeywordsColumn(DbType.MYSQL, fromColumn), targetColumn);
+        assertEquals(DataxUtils.doConvertKeywordsColumn(DbType.MYSQL, " \"`select`\" "), "`select`");


assertEquals first param is expected value, sceond param is actual value, please swap there params.

thx, I fixed

Eights-Li · 2020-02-04T09:34:48Z

...nscheduler-server/src/test/java/org/apache/dolphinscheduler/server/utils/DataxUtilsTest.java

-
-        assertEquals(DataxUtils.doConvertKeywordsColumn(DbType.MYSQL, fromColumn), targetColumn);
+        assertEquals(DataxUtils.doConvertKeywordsColumn(DbType.MYSQL, " \"`select`\" "), "`select`");
+        assertEquals(DataxUtils.doConvertKeywordsColumn(DbType.POSTGRESQL, " \"`select`\" "), "\"select\"");


assertEquals first param is expected value, sceond param is actual value, please swap there params, from 102-105 lines.

Eights-Li · 2020-02-04T09:43:20Z