Skip to content

[Feature] Add ETL function node #1856

@wen-hemin

Description

@wen-hemin

I extended the ETL node on DS, Implement the function of executing SQL for data synchronization.
Synchronize data through Datax middleware, DS need not be coupled to Datax.

I can contribute code if needed by the community.

Implementation ideas:
1.Add ETL nodes in DS
2.Generate Datax json file based on configuration parameters
3.Datax python script called by shell script task to start process

Problems with the current implementation:
1.Datax does not support multi-table synchronization, a task can only synchronize one table
2.Low resource utilization, each ETL task needs to start a JVM process

WechatIMG6269

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions