Skip to content

[Feature][DAG] dag node can't debug online #5271

@geosmart

Description

@geosmart

Now dag node can't debug online.
I have to check the yarn log/email/yarn log,or debug the code in ide to make sure the code is right, and it is Very inefficient.

Describe the feature
the dag node content(sql,pyspark,shell,python) can interactive execute to test if the code is right.

I wish

  • a sql(query) can return a sample result.
  • a sql(update) can run change count.
  • a shell/python code can show output log.
  • a spark code can get the yarn aggregate container log.

Describe the solution you'd like
pyspark can submit by apache livy, sql can submit by jdbc

Describe alternatives you've considered
juest like apache hue or webank's scriptis do

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions