[refactor](Coordinator) refactor coordinator #41730

924060929 · 2024-10-11T11:46:27Z

Proposed changes

Use NereidsSqlCoordinator instead of Coordinator because the code of Coordinator is too hard to maintaining

The main design approach is as follows:

Divide the original flat Coordinator into multiple modules, with each module maintaining high cohesion.

DistributePlanner: The logic for calculating parallelism has been extracted in [refactor](nereids) New distribute planner #36531, and in the future, we will dynamically calculate parallelism based on cost.
CoordinatorContext: Some global parameters and states related to the Coordinator are encapsulated within CoordinatorContext.
PipelineExecutionTask: The entire scheduling task is encapsulated by PipelineExecutionTask, which includes the mapping relationship between each Backend and Pipeline task. PipelineExecutionTask contains two layers of tasks, each responsible for specific duties, with state maintenance handled internally rather than being centralized in the Coordinator.
- MultiFragmentsPipelineTask: A Backend will generate multiple fragment tasks, which are bundled together and sent concurrently to the corresponding Backend.
- SingleFragmentPipelineTask: A single fragment task for a Backend.
JobProcessor: Describes two types of tasks: SQL tasks and Load tasks.
- QueryProcessor: Represents query tasks and provides a ResultReceiver to obtain query results.
- LoadProcessor: Represents Insert into and Broker load tasks, providing a blocking function to wait for load completion.
ThriftPlansBuilder: Uses the DistributedPlan structure to build thrift parameters and encapsulates some intermediate temporary variables within functions, rather than placing them in the Coordinator.

The overall Coordinator logic is more clearly organized. We can see that the NereidsCoordinator consists of only a few functions, allowing quick understanding of the main flow when reading the code.

Construct CoordinatorContext.
Enqueue the tasks.
Handle different sinks accordingly.
Register the Coordinator with QeProcessorImpl for cancellation and progress tracking.
Construct thrift parameters.
Build PipelineTask.
Initiate RPC calls to each Backend.

TODO:

delete old Coordinator
support cloud mode

doris-robot · 2024-10-11T11:46:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-10-11T11:50:58Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-11T12:04:40Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-11T12:52:48Z

clang-tidy review says "All clean, LGTM! 👍"

924060929 · 2024-10-14T06:29:26Z

run buildall

924060929 · 2024-10-14T06:32:04Z

run buildall

924060929 · 2024-10-14T09:58:40Z

run buildall

924060929 · 2024-10-14T11:49:08Z

run buildall

924060929 · 2024-10-14T11:54:42Z

run buildall

924060929 · 2024-10-14T14:16:22Z

run buildall

924060929 · 2024-10-15T02:53:28Z

run buildall

924060929 · 2024-10-15T05:49:38Z

run buildall

924060929 · 2024-10-15T08:22:35Z

run buildall

924060929 · 2024-10-15T08:27:41Z

run buildall

924060929 · 2024-10-15T12:23:55Z

run buildall

924060929 · 2024-10-15T14:23:34Z

run buildall

github-actions · 2024-10-15T14:47:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-15T15:05:20Z

TeamCity be ut coverage result:
Function Coverage: 37.46% (9709/25916)
Line Coverage: 28.75% (80635/280504)
Region Coverage: 28.19% (41708/147970)
Branch Coverage: 24.77% (21215/85638)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b0d8dee255cb2b7c2a959ef55647929ef653200e_b0d8dee255cb2b7c2a959ef55647929ef653200e/report/index.html

924060929 · 2024-10-16T13:37:56Z

run buildall

924060929 · 2024-10-16T13:40:56Z

run buildall

github-actions · 2024-10-16T13:41:49Z

clang-tidy review says "All clean, LGTM! 👍"

924060929 · 2024-10-17T07:37:29Z

run buildall

924060929 · 2024-10-17T07:38:16Z

run buildall

924060929 · 2024-11-07T12:04:08Z

run buildall

github-actions · 2024-11-07T13:52:51Z

PR approved by at least one committer and no changes requested.

…or (#43763) fix QueryProcessor cannot be cast to class LoadProcessor, introduced by #41730 Problem Summary: sql: any select statement it only meet when open debug log, so I can not write a test ``` 2024-11-12 08:15:52,266 WARN (mysql-nio-pool-0|206) [ConnectProcessor.handleQueryException():480] Process one query failed because unknown reason: java.lang.ClassCastException: class org.apache.doris.qe.runtime.QueryProcessor cannot be cast to class org.apache.doris.qe.runtime.LoadProcessor (org.apache.doris.qe.runtime.QueryProcessor and org.apache.doris.qe.runtime.LoadProcessor are in unnamed module of loader 'app') at org.apache.doris.qe.CoordinatorContext.asLoadProcessor(CoordinatorContext.java:262) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.NereidsCoordinator.getJobId(NereidsCoordinator.java:202) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.QeProcessorImpl.registerQuery(QeProcessorImpl.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1925) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1897) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:901) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:833) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:605) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:568) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:558) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:340) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:243) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:209) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:237) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:414) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ```

…or (apache#43763) fix QueryProcessor cannot be cast to class LoadProcessor, introduced by apache#41730 Problem Summary: sql: any select statement it only meet when open debug log, so I can not write a test ``` 2024-11-12 08:15:52,266 WARN (mysql-nio-pool-0|206) [ConnectProcessor.handleQueryException():480] Process one query failed because unknown reason: java.lang.ClassCastException: class org.apache.doris.qe.runtime.QueryProcessor cannot be cast to class org.apache.doris.qe.runtime.LoadProcessor (org.apache.doris.qe.runtime.QueryProcessor and org.apache.doris.qe.runtime.LoadProcessor are in unnamed module of loader 'app') at org.apache.doris.qe.CoordinatorContext.asLoadProcessor(CoordinatorContext.java:262) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.NereidsCoordinator.getJobId(NereidsCoordinator.java:202) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.QeProcessorImpl.registerQuery(QeProcessorImpl.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1925) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1897) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:901) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:833) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:605) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:568) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:558) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:340) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:243) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:209) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:237) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:414) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ```

fix new coordinator compute a wrong scanRangeNum, introduced by #41730 This bug will show a wrong progress in s3 load: ``` Progress: 0.00%(73/2147483647) ```

optimize new distribute planner performance in tpc-h, because #41730 made some performance rollback has occurred 1. fix the wrong runtime filter thrift parameters 2. not default to print distribute plan in profile, you should config `set profile_level=3` to see it 3. for shuffle join which two sides distribution of natural + execution_bucketed, support compare cost between plans of shuffle to left/right

…s CTE (#44753) fix NereidsCoordinator compute wrong result when exists CTE, introduced by #41730

…property (#47888) fix Illegal bucket shuffle join or colocate join in fragment because compute wrong join output property, introduced by #41730 the exception: ``` errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment ```

…property (apache#47888) fix Illegal bucket shuffle join or colocate join in fragment because compute wrong join output property, introduced by apache#41730 the exception: ``` errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment ```

fix colocate agg + join compute wrong result, introduced by #41730

…8934) fix colocate agg + join compute wrong result, introduced by apache#41730 (cherry picked from commit 0e5abe8)

…property (apache#47888) fix Illegal bucket shuffle join or colocate join in fragment because compute wrong join output property, introduced by apache#41730 the exception: ``` errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment ```

…8934) fix colocate agg + join compute wrong result, introduced by apache#41730

### What problem does this PR solve? Followup #50791 Add a new FE HTTP API: `/rest/v2/manager/query/statistics/trace_id`. This API will return the query runtime statistic corresponding to a given trace id. The query statistics includes info such as real-time scan rows/bytes. Internally, Doris will get query id by trace id from all Frontends, and then fetch query statistics from BE. Use pattern: 1. User set custom trace id by: `set session_context="trace_id:my_trace_id"` 2. User executes a query in same session 3. Start a http client to get query statistics in real-time during the query process. ![progress](https://github.com/user-attachments/assets/0a697c7d-d87a-4e9c-8965-c5a2d7d7836e) Also fix a bug in `CoordinatorContext.java`, to get real host. introduced from #41730 This PR also change the column name of `information_schema.processlist` table, to be same as column name in `show processlist`.

924060929 force-pushed the new_scheduler5 branch from 4314d53 to 66c693a Compare October 12, 2024 13:01

924060929 marked this pull request as ready for review October 12, 2024 13:09

924060929 force-pushed the new_scheduler5 branch from 0ec601b to 039b5e1 Compare October 15, 2024 08:27

924060929 added dev/2.1.x dev/3.0.x and removed dev/2.1.x dev/3.0.x labels Oct 17, 2024

starocean999 approved these changes Nov 7, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 7, 2024

morrySnow approved these changes Nov 7, 2024

View reviewed changes

924060929 added the not-merge/3.0 label Nov 7, 2024

924060929 merged commit 46e5294 into apache:master Nov 7, 2024

924060929 removed the not-merge/3.0 label Nov 7, 2024

924060929 mentioned this pull request Nov 12, 2024

[fix](Nereids) fix QueryProcessor cannot be cast to class LoadProcessor #43763

Merged

16 tasks

924060929 mentioned this pull request Nov 13, 2024

[fix](Nereids) fix new coordinator compute a wrong scanRangeNum #43850

Merged

16 tasks

924060929 mentioned this pull request Nov 15, 2024

[opt](Nereids) optimize performance in new distribute planner #44048

Merged

16 tasks

924060929 mentioned this pull request Nov 28, 2024

[fix](nereids) fix NereidsCoordinator compute wrong result when exists CTE #44753

Merged

16 tasks

924060929 added a commit that referenced this pull request Dec 2, 2024

[fix](nereids) fix NereidsCoordinator compute wrong result when exist…

2cde7b0

…s CTE (#44753) fix NereidsCoordinator compute wrong result when exists CTE, introduced by #41730

924060929 mentioned this pull request Feb 13, 2025

[fix](nereids) fix bucket shuffle join to right compute wrong output property #47888

Merged

16 tasks

924060929 mentioned this pull request Mar 11, 2025

[fix](nereids) fix colocate agg + join compute wrong result #48934

Merged

16 tasks

924060929 added a commit that referenced this pull request Mar 12, 2025

[fix](nereids) fix colocate agg + join compute wrong result (#48934)

0e5abe8

fix colocate agg + join compute wrong result, introduced by #41730

github-actions bot pushed a commit that referenced this pull request Mar 12, 2025

[fix](nereids) fix colocate agg + join compute wrong result (#48934)

73d81dc

fix colocate agg + join compute wrong result, introduced by #41730

924060929 deleted the new_scheduler5 branch March 25, 2025 10:25

github-actions bot pushed a commit that referenced this pull request May 27, 2025

[fix](nereids) fix colocate agg + join compute wrong result (#48934)

0b649ae

fix colocate agg + join compute wrong result, introduced by #41730

morningman mentioned this pull request Jun 2, 2025

[feat](profile) support getting query progress #51400

Merged

16 tasks

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](nereids) fix colocate agg + join compute wrong result (apache#4…

26d29e5

…8934) fix colocate agg + join compute wrong result, introduced by apache#41730

[refactor](Coordinator) refactor coordinator #41730

[refactor](Coordinator) refactor coordinator #41730

Uh oh!

Conversation

924060929 commented Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 14, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

924060929 commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

doris-robot commented Oct 15, 2024

Uh oh!

924060929 commented Oct 16, 2024

Uh oh!

924060929 commented Oct 16, 2024

Uh oh!

github-actions bot commented Oct 16, 2024

Uh oh!

924060929 commented Oct 17, 2024

Uh oh!

924060929 commented Oct 17, 2024

Uh oh!

924060929 commented Nov 7, 2024

Uh oh!

github-actions bot commented Nov 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

924060929 commented Oct 11, 2024 •

edited

Loading