Now dag node can't debug online.
I have to check the yarn log/email/yarn log,or debug the code in ide to make sure the code is right, and it is Very inefficient.
Describe the feature
the dag node content(sql,pyspark,shell,python) can interactive execute to test if the code is right.
I wish
- a sql(query) can return a sample result.
- a sql(update) can run change count.
- a shell/python code can show output log.
- a spark code can get the yarn aggregate container log.
Describe the solution you'd like
pyspark can submit by apache livy, sql can submit by jdbc
Describe alternatives you've considered
juest like apache hue or webank's scriptis do
Now dag node can't debug online.
I have to check the yarn log/email/yarn log,or debug the code in ide to make sure the code is right, and it is Very inefficient.
Describe the feature
the dag node content(sql,pyspark,shell,python) can interactive execute to test if the code is right.
I wish
Describe the solution you'd like
pyspark can submit by apache livy, sql can submit by jdbc
Describe alternatives you've considered
juest like apache hue or webank's scriptis do