[DSIP-61][Master] Refactor thread pool and state event orchestration in master by ruanwenjun · Pull Request #16327 · apache/dolphinscheduler

ruanwenjun · 2024-07-16T11:33:41Z

Purpose of the pull request

close #16423

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+        String[] processInstanceIdArray = processInstanceIds.split(Constants.COMMA);
+        List<String> errorMessage = new ArrayList<>();
+        for (String strProcessInstanceId : processInstanceIdArray) {
+            int processInstanceId = Integer.parseInt(strProcessInstanceId);


...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+            int processInstanceId = Integer.parseInt(strProcessInstanceId);
+            try {
+                execService.controlWorkflowInstance(loginUser, processInstanceId, executeType);
+                log.info("Success do action {} on workflowInstance: {}", executeType, processInstanceId);


...heduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java

+            } catch (Exception e) {
+                errorMessage.add("Failed do action " + executeType + " on workflowInstance: " + processInstanceId
+                        + "reason: " + e.getMessage());
+                log.error("Failed do action {} on workflowInstance: {}, error: {}", executeType, processInstanceId, e);


sonarqubecloud · 2024-07-16T12:33:24Z

Quality Gate failed

Failed conditions
39.7% Coverage on New Code (required ≥ 60%)

See analysis details on SonarCloud

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

ruanwenjun · 2024-08-19T12:57:55Z

Quality Gate failed

Failed conditions 7.8% Coverage on New Code (required ≥ 60%) C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

The coverage is incorrect

...hinscheduler/server/master/engine/task/lifecycle/handler/TaskStartLifecycleEventHandler.java

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java

...uler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java

SbloodyS

LGTM. That's really a huge job. 👍🏻

caishunfeng

Incomplete review

...org/apache/dolphinscheduler/api/executor/workflow/PauseWorkflowInstanceExecutorDelegate.java

...dolphinscheduler/api/executor/workflow/RecoverSuspendedWorkflowInstanceExecutorDelegate.java

.../java/org/apache/dolphinscheduler/api/executor/workflow/TriggerWorkflowExecutorDelegate.java

...n/java/org/apache/dolphinscheduler/api/executor/workflow/WorkflowInstanceControlRequest.java

...rc/main/java/org/apache/dolphinscheduler/extract/master/transportor/ITaskExecutionEvent.java

caishunfeng · 2024-08-23T09:51:31Z

...rc/main/java/org/apache/dolphinscheduler/extract/master/transportor/ITaskExecutionEvent.java

 package org.apache.dolphinscheduler.extract.master.transportor;

-public interface ITaskInstanceExecutionEvent {
+public interface ITaskExecutionEvent {


Suggest to add event source, which can help troubleshoot.

Yes, it is needed, e.g. distinguish the event from the task executor or user operation. I add todo here.

...aster/src/main/java/org/apache/dolphinscheduler/server/master/cluster/MasterSlotManager.java

...ter/src/main/java/org/apache/dolphinscheduler/server/master/engine/TaskGroupCoordinator.java

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java

...c/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEventBusFireWorker.java

...aster/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowRepository.java

...pache/dolphinscheduler/server/master/engine/command/handler/ReRunWorkflowCommandHandler.java

.../dolphinscheduler/server/master/engine/command/handler/RecoverFailureTaskCommandHandler.java

...ain/java/org/apache/dolphinscheduler/server/master/engine/graph/AbstractSuccessorParser.java

...c/main/java/org/apache/dolphinscheduler/server/master/engine/graph/WorkflowGraphFactory.java

caishunfeng · 2024-08-26T02:59:44Z

...main/java/org/apache/dolphinscheduler/server/master/engine/system/event/SystemEventType.java

+    GLOBAL_MASTER_FAILOVER,
+    MASTER_FAILOVER,


It's better to add some comments for the difference.

caishunfeng · 2024-08-26T03:04:36Z

...e/dolphinscheduler/server/master/engine/task/client/PhysicalTaskExecutorClientDelegator.java

+
+@Slf4j
+@Component
+public class PhysicalTaskExecutorClientDelegator implements ITaskExecutorClientDelegator {


Is the PhysicalTask means The tasks which execute on worker?

.../apache/dolphinscheduler/server/master/engine/task/statemachine/AbstractTaskStateAction.java

...n/java/org/apache/dolphinscheduler/server/master/runner/TaskExecutionRunnableRepository.java

...r/src/test/java/org/apache/dolphinscheduler/server/master/AbstractMasterIntegrationTest.java

sonarqubecloud · 2024-08-26T06:22:03Z

Quality Gate failed

Failed conditions
7.0% Coverage on New Code (required ≥ 60%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

caishunfeng

LGTM overall.

SbloodyS

+1

ruanwenjun requested review from SbloodyS and caishunfeng as code owners July 16, 2024 11:33

github-actions bot added backend test labels Jul 16, 2024

github-actions bot assigned ruanwenjun Jul 16, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 3 times, most recently from d29aba5 to d6052b8 Compare July 16, 2024 11:56

github-advanced-security bot found potential problems Jul 16, 2024

View reviewed changes

ruanwenjun marked this pull request as draft July 16, 2024 13:32

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from d6052b8 to 465806d Compare July 17, 2024 01:12

github-actions bot added the document label Jul 17, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 465806d to 4adedb5 Compare July 23, 2024 15:39

github-actions bot added the UI ui and front end related label Jul 23, 2024

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch from 4adedb5 to 0b40177 Compare July 25, 2024 06:41

This was referenced Jul 26, 2024

[DSIP-57][Audit] Revert PR 14981 code #16370

Merged

[DSIP-57][Audit] Refactor PR 14981 code #16367

Closed

[Improvement][Test] Add api-test cases for workflow and task #16384

Closed

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from f5a524f to c816086 Compare August 14, 2024 14:32

ruanwenjun mentioned this pull request Aug 15, 2024

[Fix-16382] Fix the bug of async master task casthread pool invocations ramp-up #16461

Merged

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from 41a627e to a101f89 Compare August 15, 2024 12:01

ruanwenjun changed the title ~~Fix serial wait workflow instance cannot stop~~ [DSIP-61][Master] Refactor thread pool and state event orchestration in master Aug 15, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 3 times, most recently from 8999a0d to 2118c1f Compare August 15, 2024 12:54

SbloodyS mentioned this pull request Aug 16, 2024

[Bug] [Process Instance ] Version 3.2.2 Workflow scheduling uses a serial wait for execution strategy, and task blocking accumulation occurs after master overloads #16474

Closed

3 tasks

ruanwenjun requested a review from Gallardot as a code owner August 19, 2024 09:22

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from 646a458 to 02044d5 Compare August 19, 2024 10:56

apache deleted a comment from sonarqubecloud bot Aug 19, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 2 times, most recently from 1ff6b24 to ddf387f Compare August 19, 2024 13:13

github-actions bot added the CI&CD label Aug 19, 2024

ruanwenjun force-pushed the dev_wenjun_fixSerialWaitWorkflowCannotKill branch 7 times, most recently from 5710bfd to be08dfb Compare August 20, 2024 14:00

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...hinscheduler/server/master/engine/task/lifecycle/handler/TaskStartLifecycleEventHandler.java Outdated Show resolved Hide resolved

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java Outdated Show resolved Hide resolved

...er-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEngine.java Outdated Show resolved Hide resolved

SbloodyS reviewed Aug 21, 2024

View reviewed changes

...uler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java Show resolved Hide resolved

SbloodyS previously approved these changes Aug 21, 2024

View reviewed changes

caishunfeng reviewed Aug 23, 2024

View reviewed changes

caishunfeng reviewed Aug 26, 2024

View reviewed changes

ruanwenjun added 3 commits August 26, 2024 13:45

[Refactor] Refactor the master threadpool and event engine

5cb97e6

Resolve comments

4433752

Fix comments from shunfeng

109d07e

caishunfeng approved these changes Aug 26, 2024

View reviewed changes

SbloodyS approved these changes Aug 26, 2024

View reviewed changes

SbloodyS mentioned this pull request Nov 21, 2024

[Bug] [dolphinscheduler - master]switch task type Endless loop #16827

Closed

3 tasks

saowu mentioned this pull request Feb 7, 2025

[Bug] [Task] A single task instance runs or runs backwards only once #16997

Open

3 tasks

Conversation

ruanwenjun commented Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Check notice

Check failure

Check failure

sonarqubecloud bot commented Jul 16, 2024

Quality Gate failed

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

ruanwenjun commented Aug 19, 2024

Quality Gate failed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

caishunfeng Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

caishunfeng Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

caishunfeng Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Aug 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 26, 2024

Quality Gate failed

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ruanwenjun commented Jul 16, 2024 •

edited

Loading