-
Notifications
You must be signed in to change notification settings - Fork 5k
[BUG] Sqoop task type can not read custom parameters #2763
Description
*For better global communication, please give priority to using English description, thx! *
Describe the bug
The custom parameters of the Sqoop task do not work, and the user's custom parameters cannot be read, for example: --verbose. Such parameters cannot be found in the Sqoop command.Currently displayed on the page, but there is no corresponding analysis on the back.
To Reproduce
Create a Sqoop type task, add any parameters in the bottom custom parameters, the custom added parameters do not take effect when the sqoop task is executed, the worker log shows that the generated sqoop command script has no custom added parameters.
Expected behavior
The custom added parameters of sqoop node should take effect
Screenshots
worker-log:

[INFO] 2020-05-21 10:30:30.185 - [taskAppId=TASK-41-1345-2808]:[51] - sqoop task params {"sourceType":"MYSQL","targetType":"HDFS","targetParams":"{"targetPath":"/tmp/sqooptest518","deleteTargetDir":true,"fileType":"--as-textfile","compressionCodec":"","fieldsTerminated":"","linesTerminated":""}","modelType":"import","sourceParams":"{"srcType":"MYSQL","srcDatasource":10,"srcTable":"im","srcQueryType":"0","srcQuerySql":"","srcColumnType":"0","srcColumns":"","srcConditionList":[],"mapColumnHive":[],"mapColumnJava":[]}","localParams":[{"prop":"--verbose","direct":"IN","type":"VARCHAR","value":""},{"prop":"--validate","direct":"IN","type":"VARCHAR","value":""}],"concurrency":1}
[INFO] 2020-05-21 10:30:30.231 - [taskAppId=TASK-41-1345-2808]:[74] - sqoop script: sqoop import -m 1 --connect jdbc:mysql://10.180.210.143:3306/test --username root --password bigdata --table im --target-dir /tmp/sqooptest518 --as-textfile --delete-target-dir --null-non-string 'NULL' --null-string 'NULL'
[INFO] 2020-05-21 10:30:30.231 - [taskAppId=TASK-41-1345-2808]:[84] - tenantCode user:hdfs, task dir:41_1345_2808
[INFO] 2020-05-21 10:30:30.232 - [taskAppId=TASK-41-1345-2808]:[89] - create command file:/tmp/dolphinscheduler/exec/process/11/41/1345/2808/41_1345_2808.command
[INFO] 2020-05-21 10:30:30.232 - [taskAppId=TASK-41-1345-2808]:[108] - command : #!/bin/sh
BASEDIR=$(cd dirname $0; pwd)
cd $BASEDIR
source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
sqoop import -m 1 --connect jdbc:mysql://10.180.210.143:3306/test --username root --password bigdata --table im --target-dir /tmp/sqooptest518 --as-textfile --delete-target-dir --null-non-string 'NULL' --null-string 'NULL'
The two parameters I added: --verbose,--validateare not displayed in the generated sqoop command script
Which version of Dolphin Scheduler:
-[dev-1.3.0],[dev]
**Requirement or improvement
For the sqoop command, because there are too many command parameters, I think it is not appropriate to directly remove the interface used for custom parameters. Hope to be able to solve this problem.