[Feature][Improvement] Support multi cluster environments - k8s type by qianli2022 · Pull Request #10096 · apache/dolphinscheduler

qianli2022 · 2022-05-18T07:46:15Z

Purpose of the pull request

Adding a cluster configuration, a cluster configuration with k8s etc. Currently only k8s multi-clusters are supported.

Brief change log

Add ClusterServiceImpl.java
Add ClusterController.java
Add ClusterMapper.java
Add cluster-modal.tsx

Verify this pull request

This change added tests and can be verified as follows:

Added ClusterControllerTest.
Added ClusterServiceTest.
Added ClusterMapperTest

close #9462

codecov-commenter · 2022-05-18T08:05:59Z

Codecov Report

Merging #10096 (b8201ce) into dev (3258438) will increase coverage by 0.32%.
The diff coverage is 83.82%.

@@             Coverage Diff              @@
##                dev   #10096      +/-   ##
============================================
+ Coverage     40.56%   40.89%   +0.32%     
- Complexity     4766     4848      +82     
============================================
  Files           877      883       +6     
  Lines         35621    35970     +349     
  Branches       3945     3991      +46     
============================================
+ Hits          14449    14709     +260     
- Misses        19737    19807      +70     
- Partials       1435     1454      +19

Impacted Files	Coverage Δ
...che/dolphinscheduler/api/k8s/K8sClientService.java	`6.38% <0.00%> (ø)`
...olphinscheduler/common/utils/ClusterConfUtils.java	`0.00% <0.00%> (ø)`
...scheduler/api/service/impl/ClusterServiceImpl.java	`88.28% <88.28%> (ø)`
...hinscheduler/api/controller/ClusterController.java	`89.47% <89.47%> (ø)`
...rg/apache/dolphinscheduler/api/dto/ClusterDto.java	`92.85% <92.85%> (ø)`
.../org/apache/dolphinscheduler/api/enums/Status.java	`100.00% <100.00%> (ø)`
...rg/apache/dolphinscheduler/dao/entity/Cluster.java	`100.00% <100.00%> (ø)`
...pache/dolphinscheduler/service/corn/CronUtils.java	`64.77% <0.00%> (-5.60%)`	⬇️
...cheduler/api/service/impl/ExecutorServiceImpl.java	`40.10% <0.00%> (-2.59%)`	⬇️
...org/apache/dolphinscheduler/remote/utils/Host.java	`37.77% <0.00%> (-2.23%)`	⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3258438...b8201ce. Read the comment docs.

kezhenxu94 · 2022-05-25T02:29:00Z

...cheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ClusterController.java

+/**
+ * cluster controller
+ * todo 这是新增的 集群环境
+ */


Remove this kind of comments, they are meaningless

Just remove the chinese comments , or all of them? I find that all other classes are commented with xx controller?

kezhenxu94 · 2022-05-25T02:30:05Z

Hi @qianli2022 , thank you, do you have an issue to link to the feature discussion?

qianli2022 · 2022-05-25T02:56:46Z

Hi @qianli2022 , thank you, do you have an issue to link to the feature discussion?
Yes, [Feature][Improvement] Support multi cluster environments](#9462)

...inscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/ClusterE2ETest.java

caishunfeng · 2022-06-08T02:50:01Z

PTAL @lenboo @zhongjiajie @songjianet

qianli2022 · 2022-06-08T05:38:26Z

I find some test classes are not counted in the unit coverage, so git show the insufficient coverage error

caishunfeng · 2022-06-09T11:13:01Z

I find some test classes are not counted in the unit coverage, so git show the insufficient coverage error

I had rerun the UT to check the test converage.

WangJPLeo

Hello, after the cluster should belong to the scope of data permissions, we will refactor function permissions and resource permissions in the next version, the purpose is to facilitate the expansion and customization of functional permissions and data permissions. You can Under the dev' branch, you can see the method of unified management of permissions in ResourcePermissionCheckService. I hope you can optimize the permission management of the cluster in the future, thank you.

caishunfeng

LGTM of the backend part.
BTW, this is a very complete pr 👍

qianli2022 · 2022-06-13T05:11:10Z

Hello, after the cluster should belong to the scope of data permissions, we will refactor function permissions and resource permissions in the next version, the purpose is to facilitate the expansion and customization of functional permissions and data permissions. You can Under the dev' branch, you can see the method of unified management of permissions in ResourcePermissionCheckService. I hope you can optimize the permission management of the cluster in the future, thank you.

OK,next pr will be namespace using multiple clusters, which will include permissions, and I will follow your latest permissions design.

dolphinscheduler-ui/src/views/security/cluster-manage/components/cluster-modal.tsx

dolphinscheduler-ui/src/views/security/cluster-manage/components/use-modal.ts

dolphinscheduler-ui/src/views/security/cluster-manage/index.tsx

use Nspace instead of css.

Remove the style.

Remove the useless file.

sonarqubecloud · 2022-06-15T03:59:57Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
27 Code Smells

88.8% Coverage
6.4% Duplication

tongwl · 2022-06-15T04:00:09Z

@Amy0104, thank you for your review, very nice catching :)

I also have some suggestions:
as we are only contributors, not owners, so our new code style is best to refer to the existing code, this will make the project code management clearer. So it would be even better if all the similar pages using the similar template, this pr front-end code is copied from the environment page. Obviously, the environment code can't be recognized by project owners, it does have some small areas that can be optimized. For our contributors, we'd better not be unconventional, but keep the same way of code writing style.

Amy0104 · 2022-06-15T04:28:42Z

Thanks for your suggestion. We will try to make the pages to a small template later.

Amy0104

LGTM.

…pache#10096) * service code * [Feature][UI] Add front-end for cluster manage * fix e2e * remove comment on cluster controller * doc * img * setting e2e.yaml * test * rerun e2e * fix bug from comment * Update index.tsx use Nspace instead of css. * Update index.tsx Remove the style. * Delete index.module.scss Remove the useless file. Co-authored-by: qianl4 <qianl4@cicso.com> Co-authored-by: William Tong <weitong@cisco.com> Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>

qianl4 and others added 2 commits May 18, 2022 14:12

service code

fe3b637

[Feature][UI] Add front-end for cluster manage

4eeaee9

qianl4 added 4 commits May 18, 2022 16:20

Merge branch 'dev' into Feature-9544-v1

3dfca2b

Merge branch 'dev' into Feature-9544-v1

f410e24

Merge branch 'dev' into Feature-9544-v1

243c45b

fix e2e

e902e1d

qianli2022 requested a review from kezhenxu94 as a code owner May 25, 2022 02:06

kezhenxu94 reviewed May 25, 2022

View reviewed changes

remove comment on cluster controller

05cdc18

qianli2022 requested a review from kezhenxu94 May 25, 2022 06:58

merge dev

a758bb0

kezhenxu94 requested review from SbloodyS and caishunfeng June 3, 2022 13:43

SbloodyS requested review from Amy0104 and songjianet June 3, 2022 14:44

caishunfeng requested a review from lenboo June 6, 2022 03:03

caishunfeng added the miss:docs missing documents in PR label Jun 6, 2022

qianl4 added 2 commits June 7, 2022 16:59

merge dev

a9487c5

doc

d5bb980

qianli2022 requested review from Tianqi-Dotes and zhongjiajie as code owners June 8, 2022 01:41

img

1578550

caishunfeng reviewed Jun 8, 2022

View reviewed changes

...inscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/ClusterE2ETest.java Show resolved Hide resolved

SbloodyS added backend e2e e2e test UI ui and front end related labels Jun 8, 2022

SbloodyS assigned qianli2022 Jun 8, 2022

caishunfeng removed the miss:docs missing documents in PR label Jun 8, 2022

setting e2e.yaml

3fbf84f

qianl4 added 3 commits June 10, 2022 10:49

merge dev

c7d0d23

test

1d12bba

rerun e2e

e5afc4f

qianli2022 requested a review from caishunfeng June 10, 2022 06:49

WangJPLeo previously approved these changes Jun 11, 2022

View reviewed changes

caishunfeng previously approved these changes Jun 13, 2022

View reviewed changes

Amy0104 requested changes Jun 13, 2022

View reviewed changes

fix bug from comment

7a6be45

tongwl dismissed stale reviews from caishunfeng and WangJPLeo via 7a6be45 June 15, 2022 02:44

Amy0104 added 3 commits June 15, 2022 11:24

Update index.tsx

ac874c4

use Nspace instead of css.

Update index.tsx

e327837

Remove the style.

Delete index.module.scss

b8201ce

Remove the useless file.

Amy0104 approved these changes Jun 15, 2022

View reviewed changes

Amy0104 added this to the 3.1.0-alpha milestone Jun 15, 2022

Amy0104 merged commit ff065d8 into apache:dev Jun 15, 2022

Conversation

qianli2022 commented May 18, 2022 • edited by SbloodyS Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

codecov-commenter commented May 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kezhenxu94 May 25, 2022

Choose a reason for hiding this comment

Uh oh!

qianli2022 May 25, 2022

Choose a reason for hiding this comment

Uh oh!

kezhenxu94 commented May 25, 2022

Uh oh!

qianli2022 commented May 25, 2022

Uh oh!

Uh oh!

caishunfeng commented Jun 8, 2022

Uh oh!

qianli2022 commented Jun 8, 2022

Uh oh!

caishunfeng commented Jun 9, 2022

Uh oh!

WangJPLeo left a comment

Choose a reason for hiding this comment

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

qianli2022 commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 15, 2022

Uh oh!

tongwl commented Jun 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amy0104 commented Jun 15, 2022

Uh oh!

Amy0104 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

qianli2022 commented May 18, 2022 •

edited by SbloodyS

Loading

codecov-commenter commented May 18, 2022 •

edited

Loading

qianli2022 commented Jun 13, 2022 •

edited

Loading

tongwl commented Jun 15, 2022 •

edited

Loading