Add Python API implementation of workflows-as-code by zhongjiajie · Pull Request #6269 · apache/dolphinscheduler

zhongjiajie · 2021-09-18T08:00:19Z

Purpose of the pull request

First commit and init for pydolphinscheduler

Brief change log

Initialization pydolphinscheduler, a python sdk for DolphinScheduler which could define workflow by code

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java

zhongjiajie · 2021-09-23T14:37:13Z

How to reproduce

Start java gateway: run dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java in your IDEA
Run python tutorial: run python SDK tutorial in https://github.com/apache/dolphinscheduler/pull/6269/files#diff-5561fec6b57cc611bee2b0d8f030965d76bdd202801d9f8a1e2e74c21769bc41

After this two steps, you could see above error and found server are null

zhongjiajie · 2021-09-24T11:06:20Z

FYI, after I add commit a4d84aa
we could install python package pydolphinscheduler locally by run command python setup.py install in dir dolphinscheduler/dolphinscheduler-python/pydolphinscheduler

After command finish, we could run tutorial.py directly by

cd dolphinscheduler-python/pydolphinscheduler
python example/tutorial.py

to make a quick look how pydolphinscheduler work

zhongjiajie · 2021-09-24T11:08:40Z

cc: @caishunfeng

caishunfeng · 2021-09-24T12:50:53Z

The PythonGatewayServer is run by spring and it can manage beans by annotations like @Autowired,
so if you new the PythonGatewayServer by yourself, which is out of Spring control, you will get some null beans.

you can have a try like GatewayServer server = new GatewayServer(this); instead of new PythonGatewayServer.

zhongjiajie

Some suggestion for you @caishunfeng

...inscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java

zhongjiajie · 2021-09-24T14:28:54Z

@caishunfeng FYI, I add you as this PR co-author manually in commit 9da1811

caishunfeng · 2021-09-25T01:51:47Z

Some suggestion for you @caishunfeng

Thanks, that's a good suggestion

zhongjiajie · 2021-09-29T12:33:44Z

After commit 37bd41a add, we fix error while adding new task and change task attribute(including task name).

But I found out a bug when update workflow tasks, I create an new issue to track it #6418

zhongjiajie · 2021-09-29T13:39:50Z

Could I code style check and auto fix in my local env?

zhongjiajie · 2021-09-29T13:55:22Z

Could I code style check and auto fix in my local env?

I found out we have check-style file and idea plugin in https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html

codecov-commenter · 2021-09-30T07:44:29Z

Codecov Report

Merging #6269 (67f956a) into dev (d91801e) will decrease coverage by 0.08%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev    #6269      +/-   ##
============================================
- Coverage     38.55%   38.46%   -0.09%     
+ Complexity     3224     3221       -3     
============================================
  Files           646      646              
  Lines         25840    25866      +26     
  Branches       2799     2804       +5     
============================================
- Hits           9963     9950      -13     
- Misses        14961    15002      +41     
+ Partials        916      914       -2

Impacted Files	Coverage Δ
...scheduler/api/service/impl/ProjectServiceImpl.java	`74.03% <0.00%> (-3.88%)`	⬇️
...inscheduler/api/service/impl/QueueServiceImpl.java	`64.81% <0.00%> (-7.36%)`	⬇️
...nscheduler/api/service/impl/TenantServiceImpl.java	`70.27% <0.00%> (-4.02%)`	⬇️
.../org/apache/dolphinscheduler/common/Constants.java	`75.00% <ø> (ø)`
...ache/dolphinscheduler/server/log/LoggerServer.java	`78.26% <0.00%> (-8.70%)`	⬇️
...r/plugin/registry/zookeeper/ZookeeperRegistry.java	`39.00% <0.00%> (-5.68%)`	⬇️
...er/master/processor/queue/TaskResponseService.java	`57.31% <0.00%> (-3.66%)`	⬇️
...dolphinscheduler/remote/future/ResponseFuture.java	`81.35% <0.00%> (-1.70%)`	⬇️
...er/master/dispatch/host/assign/RandomSelector.java	`83.33% <0.00%> (+5.55%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d91801e...67f956a. Read the comment docs.

zhongjiajie · 2021-09-30T08:05:13Z

Questions: could I add py4j.jar to file known-dependencies.txt directly? for passing check-LICENSE CI? py4j license in https://github.com/bartdag/py4j/blob/master/LICENSE.txt

kezhenxu94 · 2021-09-30T09:34:40Z

Questions: could I add py4j.jar to file known-dependencies.txt directly? for passing check-LICENSE CI? py4j license in https://github.com/bartdag/py4j/blob/master/LICENSE.txt

You need to check whether the jar is license-compatible with APL2.0, and add the corresponding license info into the dolphinscheduler-dist/release-docs/licenses.

zhongjiajie · 2021-10-27T09:48:32Z

You need to check whether the jar is license-compatible with APL2.0

@kezhenxu94 How could I check whether it compatible to APL2.0 or not?

kezhenxu94 · 2021-10-27T09:57:49Z

You need to check whether the jar is license-compatible with APL2.0

@kezhenxu94 How could I check whether it compatible to APL2.0 or not?

Read this https://www.apache.org/legal/resolved.html

According to your comment, py4j is BSD 2.0 so it's compatible with APL2.0

zhongjiajie · 2021-10-27T10:07:36Z

Thanks, BTW your response as fast as usual

zhongjiajie · 2021-10-27T10:47:44Z

It's good to go now, I add example to desc how it work, and add some doc about pydolphinscheduler dev env build and core concept in readme
PTAL @dailidong @CalvinKirs @lenboo @caishunfeng

davidzollo

great job, such a big feature
+1

kezhenxu94

You need to also add the new dependency license to https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-dist/release-docs/LICENSE

caishunfeng · 2021-10-28T01:51:47Z

dolphinscheduler-python/pom.xml

+    <dependency>
+        <groupId>net.sf.py4j</groupId>
+        <artifactId>py4j</artifactId>
+        <version>0.10.9</version>


the dependency should put in parent pom.xml to manage version easily.

I add this to parent pom.xml, but it seem we should still add dependency

<dependency> <groupId>net.sf.py4j</groupId> <artifactId>py4j</artifactId> </dependency>

to python/pom.xml. otherwise it could not find py4j dependent. Am I wrong? @caishunfeng

dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/java_gateway.py

zhongjiajie · 2021-10-28T03:06:06Z

You need to also add the new dependency license to https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-dist/release-docs/LICENSE

I add it to release-docs/LICENSE now in 2da0790, thinks

zhongjiajie · 2021-10-28T03:22:45Z

When I run python define code, I could see workflow was created and process instance start but failed. The screenshot as below

And I see some error log look like

2021-10-28 11:17:07.160  INFO 30083 --- [tp905404580-123] o.a.d.a.service.impl.LoggerServiceImpl   : log host : 192.168.1.102 , logPath :  , logServer port : 50051
2021-10-28 11:17:07.160  INFO 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService       : roll view log, host : 192.168.1.102, port : 50051, path , skipLineNum 0 ,limit 1000
2021-10-28 11:17:07.193  WARN 30083 --- [tp905404580-123] o.a.d.remote.NettyRemotingClient         : connect to Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} error

zhongjiajie · 2021-10-28T07:46:54Z

When I run python define code, I could see workflow was created and process instance start but failed. The screenshot as below

And I see some error log look like

2021-10-28 11:17:07.160  INFO 30083 --- [tp905404580-123] o.a.d.a.service.impl.LoggerServiceImpl   : log host : 192.168.1.102 , logPath :  , logServer port : 50051
2021-10-28 11:17:07.160  INFO 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService       : roll view log, host : 192.168.1.102, port : 50051, path , skipLineNum 0 ,limit 1000
2021-10-28 11:17:07.193  WARN 30083 --- [tp905404580-123] o.a.d.remote.NettyRemotingClient         : connect to Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} error

After I inspect it, I find it cause due to my laptop cpu load too high and work do not available and task mark failed. Maybe we should add some notice for this?

But for this PR, I think is ready to go now.

zhongjiajie · 2021-10-28T07:52:14Z

PTAL @dailidong @lenboo @CalvinKirs @caishunfeng

sonarqubecloud · 2021-10-28T09:26:53Z

SonarCloud Quality Gate failed.

2 Bugs
0 Vulnerabilities
0 Security Hotspots
13 Code Smells

0.0% Coverage
0.0% Duplication

caishunfeng · 2021-10-28T09:41:29Z

+1

CalvinKirs

License good.

lenboo

+1

zhongjiajie force-pushed the init-pyds branch from 0626870 to bebbd0d Compare September 18, 2021 08:04

zhongjiajie changed the title ~~Init DS python SDK pydolphinscheduler~~ [WIP] Init DS python SDK pydolphinscheduler Sep 18, 2021

zhongjiajie commented Sep 23, 2021

View reviewed changes

dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java Outdated Show resolved Hide resolved

zhongjiajie commented Sep 24, 2021

View reviewed changes

...inscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java Outdated Show resolved Hide resolved

zhongjiajie force-pushed the init-pyds branch from 9da1811 to 9eca9f5 Compare September 26, 2021 02:10

zhongjiajie changed the title ~~[WIP] Init DS python SDK pydolphinscheduler~~ [WIP] Add Python API implementation of workflows-as-code Sep 28, 2021

zhongjiajie mentioned this pull request Sep 28, 2021

Remove unused parameter in function genTaskCodeList #6395

Merged

zhongjiajie force-pushed the init-pyds branch from 3ad58c6 to 6a064d1 Compare September 28, 2021 08:39

zhongjiajie mentioned this pull request Sep 29, 2021

[DSIP-1][Feature][Parent] Add Python API for DolphinScheduler #6407

Closed

30 tasks

zhongjiajie force-pushed the init-pyds branch from 9f7b5c5 to 18b3157 Compare September 30, 2021 07:22

zhongjiajie added 4 commits October 27, 2021 17:38

Init DS python SDK pydolphinscheduler: python code definition

4352c26

Doc first

91cae36

Add quick start and developer doc

0ec7237

Java documentation change

fa7629e

zhongjiajie force-pushed the init-pyds branch from 4410c04 to fa7629e Compare October 27, 2021 10:08

Add LICENSE-py4j.txt

4462589

zhongjiajie marked this pull request as ready for review October 27, 2021 10:10

zhongjiajie changed the title ~~[WIP] Add Python API implementation of workflows-as-code~~ Add Python API implementation of workflows-as-code Oct 27, 2021

davidzollo previously approved these changes Oct 27, 2021

View reviewed changes

kezhenxu94 reviewed Oct 27, 2021

View reviewed changes

caishunfeng reviewed Oct 28, 2021

View reviewed changes

dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/java_gateway.py Outdated Show resolved Hide resolved

zhongjiajie added 3 commits October 28, 2021 10:15

Add py4j to release-docs/LICENSE

2da0790

Move dependency version to parent pom

4d13342

Remove outdated code

67f956a

zhongjiajie dismissed davidzollo’s stale review via 67f956a October 28, 2021 03:01

zhongjiajie requested a review from davidzollo October 28, 2021 07:52

Add tenant parameter to tutorial

f358e59

CalvinKirs approved these changes Oct 28, 2021

View reviewed changes

lenboo approved these changes Oct 31, 2021

View reviewed changes

lenboo merged commit dd6ed36 into apache:dev Oct 31, 2021

zhongjiajie deleted the init-pyds branch November 1, 2021 01:29

Conversation

zhongjiajie commented Sep 18, 2021

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

Uh oh!

zhongjiajie commented Sep 23, 2021

How to reproduce

Uh oh!

zhongjiajie commented Sep 24, 2021

Uh oh!

zhongjiajie commented Sep 24, 2021

Uh oh!

caishunfeng commented Sep 24, 2021

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhongjiajie commented Sep 24, 2021

Uh oh!

caishunfeng commented Sep 25, 2021

Uh oh!

zhongjiajie commented Sep 29, 2021

Uh oh!

zhongjiajie commented Sep 29, 2021

Uh oh!

zhongjiajie commented Sep 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zhongjiajie commented Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kezhenxu94 commented Sep 30, 2021

Uh oh!

zhongjiajie commented Oct 27, 2021

Uh oh!

kezhenxu94 commented Oct 27, 2021

Uh oh!

zhongjiajie commented Oct 27, 2021

Uh oh!

zhongjiajie commented Oct 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidzollo left a comment

Choose a reason for hiding this comment

Uh oh!

kezhenxu94 left a comment

Choose a reason for hiding this comment

Uh oh!

caishunfeng Oct 28, 2021

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 28, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhongjiajie commented Oct 28, 2021

Uh oh!

zhongjiajie commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhongjiajie commented Oct 28, 2021

Uh oh!

zhongjiajie commented Oct 28, 2021

Uh oh!

sonarqubecloud bot commented Oct 28, 2021

Uh oh!

caishunfeng commented Oct 28, 2021

Uh oh!

CalvinKirs left a comment

Choose a reason for hiding this comment

Uh oh!

lenboo left a comment

Choose a reason for hiding this comment

zhongjiajie commented Sep 29, 2021 •

edited

Loading

codecov-commenter commented Sep 30, 2021 •

edited

Loading

zhongjiajie commented Sep 30, 2021 •

edited

Loading

zhongjiajie commented Oct 27, 2021 •

edited

Loading

zhongjiajie commented Oct 28, 2021 •

edited

Loading