Skip to content

[BUG] Sqoop task type can not read custom parameters #2763

@743294668

Description

@743294668

*For better global communication, please give priority to using English description, thx! *

Describe the bug
The custom parameters of the Sqoop task do not work, and the user's custom parameters cannot be read, for example: --verbose. Such parameters cannot be found in the Sqoop command.Currently displayed on the page, but there is no corresponding analysis on the back.

To Reproduce
Create a Sqoop type task, add any parameters in the bottom custom parameters, the custom added parameters do not take effect when the sqoop task is executed, the worker log shows that the generated sqoop command script has no custom added parameters.

Expected behavior
The custom added parameters of sqoop node should take effect

Screenshots
worker-log:
image
[INFO] 2020-05-21 10:30:30.185 - [taskAppId=TASK-41-1345-2808]:[51] - sqoop task params {"sourceType":"MYSQL","targetType":"HDFS","targetParams":"{"targetPath":"/tmp/sqooptest518","deleteTargetDir":true,"fileType":"--as-textfile","compressionCodec":"","fieldsTerminated":"","linesTerminated":""}","modelType":"import","sourceParams":"{"srcType":"MYSQL","srcDatasource":10,"srcTable":"im","srcQueryType":"0","srcQuerySql":"","srcColumnType":"0","srcColumns":"","srcConditionList":[],"mapColumnHive":[],"mapColumnJava":[]}","localParams":[{"prop":"--verbose","direct":"IN","type":"VARCHAR","value":""},{"prop":"--validate","direct":"IN","type":"VARCHAR","value":""}],"concurrency":1}
[INFO] 2020-05-21 10:30:30.231 - [taskAppId=TASK-41-1345-2808]:[74] - sqoop script: sqoop import -m 1 --connect jdbc:mysql://10.180.210.143:3306/test --username root --password bigdata --table im --target-dir /tmp/sqooptest518 --as-textfile --delete-target-dir --null-non-string 'NULL' --null-string 'NULL'
[INFO] 2020-05-21 10:30:30.231 - [taskAppId=TASK-41-1345-2808]:[84] - tenantCode user:hdfs, task dir:41_1345_2808
[INFO] 2020-05-21 10:30:30.232 - [taskAppId=TASK-41-1345-2808]:[89] - create command file:/tmp/dolphinscheduler/exec/process/11/41/1345/2808/41_1345_2808.command
[INFO] 2020-05-21 10:30:30.232 - [taskAppId=TASK-41-1345-2808]:[108] - command : #!/bin/sh
BASEDIR=$(cd dirname $0; pwd)
cd $BASEDIR
source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
sqoop import -m 1 --connect jdbc:mysql://10.180.210.143:3306/test --username root --password bigdata --table im --target-dir /tmp/sqooptest518 --as-textfile --delete-target-dir --null-non-string 'NULL' --null-string 'NULL'

The two parameters I added: --verbose,--validateare not displayed in the generated sqoop command script

Which version of Dolphin Scheduler:
-[dev-1.3.0],[dev]

**Requirement or improvement
For the sqoop command, because there are too many command parameters, I think it is not appropriate to directly remove the interface used for custom parameters. Hope to be able to solve this problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions