-
Notifications
You must be signed in to change notification settings - Fork 5k
[Feature][DAG] dag node can't debug online #5271
Copy link
Copy link
Closed
Labels
Description
Now dag node can't debug online.
I have to check the yarn log/email/yarn log,or debug the code in ide to make sure the code is right, and it is Very inefficient.
Describe the feature
the dag node content(sql,pyspark,shell,python) can interactive execute to test if the code is right.
I wish
- a sql(query) can return a sample result.
- a sql(update) can run change count.
- a shell/python code can show output log.
- a spark code can get the yarn aggregate container log.
Describe the solution you'd like
pyspark can submit by apache livy, sql can submit by jdbc
Describe alternatives you've considered
juest like apache hue or webank's scriptis do
Reactions are currently unavailable