[DSIP-92][Master] Refactor workflow serial strategy by ruanwenjun · Pull Request #17531 · apache/dolphinscheduler

ruanwenjun · 2025-09-22T14:31:34Z

Purpose of the pull request

close #17703

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

.../apache/dolphinscheduler/server/master/engine/workflow/serial/WorkflowSerialCoordinator.java

...va/org/apache/dolphinscheduler/server/master/engine/workflow/serial/SerialCommandsGroup.java

.github/workflows/backend.yml

dolphinscheduler-dao/src/main/resources/sql/upgrade/3.4.0_schema/mysql/dolphinscheduler_ddl.sql

SbloodyS

Gernerally LGTM.

dolphinscheduler-dao/src/main/resources/sql/upgrade/3.4.0_schema/mysql/dolphinscheduler_dml.sql

...eduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/SerialCommandMapper.xml

ruanwenjun · 2025-11-25T14:41:59Z

@copilot

Copilot

Pull request overview

This pull request implements a refactored workflow serial execution strategy (DSIP-92) by introducing a dedicated serial command queue mechanism to manage workflows with serial execution types (SERIAL_WAIT, SERIAL_DISCARD, SERIAL_PRIORITY).

Key Changes:

Introduces a new database table t_ds_serial_command to queue serial workflow commands
Refactors workflow trigger classes to return ImmutablePair<WorkflowDefinition, WorkflowInstance> instead of just WorkflowInstance, enabling execution type checks during triggering
Implements WorkflowSerialCoordinator to manage serial command queue processing with dedicated handlers for each serial execution strategy
Updates workflow state management to handle SERIAL_WAIT status for workflows awaiting serial execution
Removes unused command comment and deprecated method references

Reviewed changes

Copilot reviewed 47 out of 49 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
dolphinscheduler-worker/.../PhysicalTaskExecutor.java	Removed newline character from log message formatting
dolphinscheduler-master/.../trigger/WorkflowScheduleTrigger.java	Updated return type to ImmutablePair and added ImmutablePair import
dolphinscheduler-master/.../trigger/WorkflowManualTrigger.java	Updated return type to ImmutablePair and added ImmutablePair import
dolphinscheduler-master/.../trigger/WorkflowBackfillTrigger.java	Updated return type to ImmutablePair and added ImmutablePair import
dolphinscheduler-master/.../trigger/AbstractWorkflowTrigger.java	Refactored to support serial vs parallel execution, added SerialCommandDao
dolphinscheduler-master/.../trigger/AbstractWorkflowInstanceTrigger.java	Added serial execution handling for workflow instance triggers
dolphinscheduler-master/.../trigger/WorkflowInstanceConstructor.java	New empty interface (appears to be placeholder)
dolphinscheduler-master/.../serial/WorkflowSerialCoordinator.java	New coordinator for managing serial workflow command queue
dolphinscheduler-master/.../serial/SerialCommandsGroup.java	New DTO for grouping serial commands by workflow definition
dolphinscheduler-master/.../serial/SerialCommandWaitHandler.java	Handler for SERIAL_WAIT execution type
dolphinscheduler-master/.../serial/SerialCommandPriorityHandler.java	Handler for SERIAL_PRIORITY execution type
dolphinscheduler-master/.../serial/SerialCommandDiscardHandler.java	Handler for SERIAL_DISCARD execution type
dolphinscheduler-master/.../serial/ISerialCommandHandler.java	Interface for serial command handlers
dolphinscheduler-master/.../serial/AbstractSerialCommandHandler.java	Base implementation for serial command handlers
dolphinscheduler-master/.../statemachine/AbstractWorkflowStateAction.java	Added transaction support for workflow finalization with serial command cleanup
dolphinscheduler-master/.../trigger/SubWorkflowManualTrigger.java	Updated to work with ImmutablePair return type
dolphinscheduler-master/.../handler/WorkflowFailoverCommandHandler.java	Removed unused ITaskGroupCoordinator import
dolphinscheduler-master/.../handler/RunWorkflowCommandHandler.java	Removed unused imports
dolphinscheduler-master/.../handler/RecoverSerialWaitCommandHandler.java	New handler for RECOVER_SERIAL_WAIT command type
dolphinscheduler-master/.../handler/ReRunWorkflowCommandHandler.java	Removed unused ApplicationContext import
dolphinscheduler-master/.../engine/TaskGroupCoordinator.java	Added thread name for TaskGroupCoordinator
dolphinscheduler-master/.../engine/IWorkflowSerialCoordinator.java	New interface for workflow serial coordinator
dolphinscheduler-dao/.../sql/.../postgresql/dolphinscheduler_dml.sql	Data migration for SSH datasource parameter fix
dolphinscheduler-dao/.../sql/.../postgresql/dolphinscheduler_ddl.sql	Added t_ds_serial_command table and task group queue indexes
dolphinscheduler-dao/.../sql/.../mysql/dolphinscheduler_dml.sql	Data migration for SSH datasource parameter fix
dolphinscheduler-dao/.../sql/.../mysql/dolphinscheduler_ddl.sql	Added t_ds_serial_command table and task group queue indexes
dolphinscheduler-dao/.../sql/dolphinscheduler_postgresql.sql	Added t_ds_serial_command table definition
dolphinscheduler-dao/.../sql/dolphinscheduler_mysql.sql	Added t_ds_serial_command table definition
dolphinscheduler-dao/.../sql/dolphinscheduler_h2.sql	Added t_ds_serial_command table definition
dolphinscheduler-dao/.../mapper/SerialCommandMapper.xml	MyBatis XML mapper for serial command operations
dolphinscheduler-dao/.../impl/WorkflowSerialQueueDaoImpl.java	DAO implementation for WorkflowSerialQueue (appears unused)
dolphinscheduler-dao/.../impl/WorkflowInstanceDaoImpl.java	Removed unused performTransactionalUpsert method
dolphinscheduler-dao/.../impl/SerialCommandDaoImpl.java	DAO implementation for SerialCommand operations
dolphinscheduler-dao/.../repository/WorkflowInstanceDao.java	Removed performTransactionalUpsert method from interface
dolphinscheduler-dao/.../repository/SerialCommandDao.java	New DAO interface for SerialCommand
dolphinscheduler-dao/.../repository/IWorkflowSerialQueueDao.java	New DAO interface for WorkflowSerialQueue (appears unused)
dolphinscheduler-dao/.../model/SerialCommandDto.java	DTO for serial command with state enum
dolphinscheduler-dao/.../mapper/WorkflowSerialQueueMapper.java	MyBatis mapper for WorkflowSerialQueue (appears unused)
dolphinscheduler-dao/.../mapper/SerialCommandMapper.java	MyBatis mapper interface for SerialCommand
dolphinscheduler-dao/.../entity/WorkflowSerialQueue.java	Entity for workflow serial queue (appears unused, incorrect table name)
dolphinscheduler-dao/.../entity/WorkflowInstance.java	Removed documentation comment for setStateWithDesc method
dolphinscheduler-dao/.../entity/SerialCommand.java	New entity for t_ds_serial_command table
dolphinscheduler-common/.../enums/WorkflowExecutionTypeEnum.java	Added isSerial() helper method
dolphinscheduler-common/.../enums/CommandType.java	Removed TODO comment about RECOVER_SERIAL_WAIT
dolphinscheduler-api/.../StopWorkflowInstanceExecutorDelegate.java	Added transaction support for stopping with serial command cleanup
dolphinscheduler-api/.../PauseWorkflowInstanceExecutorDelegate.java	Added transaction support for pausing with serial command cleanup
dolphinscheduler-api/.../dto/workflow/WorkflowCreateRequest.java	Added default value PARALLEL for executionType field
docs/docs/en/guide/upgrade/incompatible.md	Reorganized version sections and added t_ds_serial_command entry
.github/workflows/backend.yml	Added version 3.3.1 to schema check matrix

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql

dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql

docs/docs/en/guide/upgrade/incompatible.md

.../apache/dolphinscheduler/server/master/engine/workflow/serial/WorkflowSerialCoordinator.java

...nscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/WorkflowSerialQueue.java

...ache/dolphinscheduler/server/master/engine/workflow/trigger/WorkflowInstanceConstructor.java

.../apache/dolphinscheduler/server/master/engine/workflow/serial/WorkflowSerialCoordinator.java

dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql

dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql

.../dolphinscheduler/server/master/engine/workflow/trigger/AbstractWorkflowInstanceTrigger.java

sonarqubecloud · 2025-11-27T09:04:36Z

Quality Gate passed

Issues
40 New issues
0 Accepted issues

Measures
0 Security Hotspots
60.8% Coverage on New Code
1.3% Duplication on New Code

See analysis details on SonarQube Cloud

ruanwenjun requested review from EricGao888, SbloodyS and caishunfeng as code owners September 22, 2025 14:31

ruanwenjun marked this pull request as draft September 22, 2025 14:31

github-actions bot added the backend label Sep 22, 2025

github-actions bot assigned ruanwenjun Sep 22, 2025

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch 2 times, most recently from 934e24d to 6f992ab Compare September 25, 2025 13:48

github-advanced-security bot found potential problems Sep 25, 2025

View reviewed changes

.../apache/dolphinscheduler/server/master/engine/workflow/serial/WorkflowSerialCoordinator.java Fixed Show fixed Hide fixed

.../apache/dolphinscheduler/server/master/engine/workflow/serial/WorkflowSerialCoordinator.java Fixed Show fixed Hide fixed

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch 2 times, most recently from 7c6aa80 to ab93648 Compare October 16, 2025 09:19

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch 2 times, most recently from 625e396 to 0aebdb1 Compare October 30, 2025 15:18

github-actions bot added the CI&CD label Oct 30, 2025

ruanwenjun added this to the 3.4.0 milestone Oct 30, 2025

github-advanced-security bot found potential problems Oct 30, 2025

View reviewed changes

...va/org/apache/dolphinscheduler/server/master/engine/workflow/serial/SerialCommandsGroup.java Dismissed Show dismissed Hide dismissed

SbloodyS mentioned this pull request Nov 3, 2025

[Bug] Serial discards not working. #17636

Closed

3 tasks

SbloodyS modified the milestone: 3.4.0 Nov 11, 2025

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch 3 times, most recently from 0f8ac4d to 86830db Compare November 19, 2025 02:10

ruanwenjun changed the title ~~Support workflow serial strategy~~ [DSIP-92][Master] Refactor workflow serial strategy Nov 19, 2025

ruanwenjun marked this pull request as ready for review November 19, 2025 02:18

ruanwenjun commented Nov 19, 2025

View reviewed changes

.github/workflows/backend.yml Outdated Show resolved Hide resolved

SbloodyS added the DSIP label Nov 19, 2025

SbloodyS reviewed Nov 20, 2025

View reviewed changes

dolphinscheduler-dao/src/main/resources/sql/upgrade/3.4.0_schema/mysql/dolphinscheduler_ddl.sql Show resolved Hide resolved

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch from 86830db to 9113a61 Compare November 20, 2025 14:02

github-actions bot added the document label Nov 20, 2025

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch from 9113a61 to 06858a0 Compare November 20, 2025 14:42

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch from 06858a0 to fd369b7 Compare November 20, 2025 14:47

SbloodyS reviewed Nov 24, 2025

View reviewed changes

dolphinscheduler-dao/src/main/resources/sql/upgrade/3.4.0_schema/mysql/dolphinscheduler_dml.sql Show resolved Hide resolved

...eduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/SerialCommandMapper.xml Show resolved Hide resolved

ruanwenjun requested a review from Copilot November 25, 2025 14:50

Copilot started reviewing on behalf of ruanwenjun November 25, 2025 14:51 View session

Copilot finished reviewing on behalf of ruanwenjun November 25, 2025 14:55

Copilot AI reviewed Nov 25, 2025

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch 2 times, most recently from 6484426 to 878f3c8 Compare November 25, 2025 15:29

ruanwenjun added 2 commits November 27, 2025 16:17

Support workflow serial strategy

998e721

Add IT case

f7401f4

github-actions bot added the test label Nov 27, 2025

ruanwenjun force-pushed the dev_wenjun_supportSerialStrategy branch from 688d088 to f7401f4 Compare November 27, 2025 08:18

zhongjiajie approved these changes Nov 27, 2025

View reviewed changes

SbloodyS approved these changes Nov 27, 2025

View reviewed changes

ruanwenjun merged commit 40ed67f into apache:dev Nov 27, 2025
75 checks passed

ruanwenjun deleted the dev_wenjun_supportSerialStrategy branch November 27, 2025 15:00

Conversation

ruanwenjun commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ruanwenjun commented Nov 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes:

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Nov 27, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ruanwenjun commented Sep 22, 2025 •

edited

Loading