Skip to content

[opt](nereids)move runtime filter info from instance level to BE level #56978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 1 commit into from
Oct 16, 2025
Merged

[opt](nereids)move runtime filter info from instance level to BE level #56978

englefly merged 1 commit into from
Oct 16, 2025

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Oct 15, 2025

What problem does this PR solve?

Runtime filter info is shared among all instances within a BE.
This change saves thrift transfering effort.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@englefly
Copy link
Contributor Author

englefly commented Oct 15, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.27% (1638/1967)
Line Coverage 67.86% (28934/42638)
Region Coverage 68.09% (14257/20937)
Branch Coverage 58.41% (7591/12996)

@doris-robot
Copy link

doris-robot commented Oct 15, 2025

TPC-DS: Total hot run time: 188193 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a18aabdeebb19f8557ee5ee5acce618f581a7b1d, data reload: false

query1	1058	446	405	405
query2	6560	1694	1695	1694
query3	6761	222	221	221
query4	26619	23828	23650	23650
query5	4613	659	502	502
query6	346	260	229	229
query7	4671	505	301	301
query8	308	296	264	264
query9	8738	2564	2555	2555
query10	515	358	301	301
query11	15376	15058	15666	15058
query12	215	138	132	132
query13	1847	684	477	477
query14	11594	9713	9603	9603
query15	346	200	190	190
query16	8169	699	531	531
query17	2361	780	651	651
query18	2235	430	342	342
query19	310	278	201	201
query20	317	184	145	145
query21	212	135	114	114
query22	4586	4920	4641	4641
query23	35145	34175	33423	33423
query24	8088	2339	2390	2339
query25	566	510	437	437
query26	1224	268	156	156
query27	2680	496	353	353
query28	4118	2138	2152	2138
query29	776	611	479	479
query30	295	220	192	192
query31	916	826	751	751
query32	77	66	65	65
query33	555	369	320	320
query34	800	860	519	519
query35	796	826	744	744
query36	932	1004	910	910
query37	124	113	81	81
query38	3576	3601	3461	3461
query39	1533	1411	1424	1411
query40	217	125	120	120
query41	63	59	60	59
query42	118	107	113	107
query43	482	487	473	473
query44	1312	822	814	814
query45	185	183	168	168
query46	844	988	643	643
query47	1765	1809	1727	1727
query48	397	422	325	325
query49	755	525	424	424
query50	648	688	411	411
query51	3939	3933	3920	3920
query52	113	108	97	97
query53	228	270	200	200
query54	608	586	522	522
query55	92	82	84	82
query56	342	326	294	294
query57	1206	1186	1139	1139
query58	292	267	301	267
query59	2491	2703	2521	2521
query60	343	332	318	318
query61	160	154	151	151
query62	814	738	659	659
query63	231	198	193	193
query64	4218	1154	823	823
query65	4104	3969	3965	3965
query66	1069	459	338	338
query67	15572	15581	15270	15270
query68	8775	941	643	643
query69	469	315	284	284
query70	1377	1338	1306	1306
query71	477	344	308	308
query72	5968	2612	5310	2612
query73	652	775	368	368
query74	9097	9237	8659	8659
query75	4256	3352	2841	2841
query76	3380	1155	738	738
query77	932	405	316	316
query78	9691	9928	8858	8858
query79	2030	844	591	591
query80	641	566	487	487
query81	515	268	234	234
query82	388	174	136	136
query83	273	279	253	253
query84	258	113	99	99
query85	993	462	418	418
query86	358	307	311	307
query87	3750	3789	3695	3695
query88	3830	2245	2232	2232
query89	406	334	298	298
query90	1919	218	220	218
query91	170	166	131	131
query92	93	70	64	64
query93	1822	984	645	645
query94	686	429	339	339
query95	401	322	315	315
query96	493	587	289	289
query97	2971	2964	2890	2890
query98	242	234	219	219
query99	1468	1458	1384	1384
Total cold run time: 281347 ms
Total hot run time: 188193 ms

@doris-robot
Copy link

doris-robot commented Oct 15, 2025

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a18aabdeebb19f8557ee5ee5acce618f581a7b1d, data reload: false

query1	0.05	0.05	0.04
query2	0.12	0.06	0.06
query3	0.30	0.07	0.07
query4	1.61	0.09	0.09
query5	0.27	0.25	0.25
query6	1.17	0.64	0.65
query7	0.03	0.02	0.02
query8	0.07	0.05	0.05
query9	0.68	0.54	0.53
query10	0.58	0.59	0.59
query11	0.25	0.14	0.13
query12	0.26	0.14	0.14
query13	0.65	0.64	0.64
query14	1.04	1.04	1.07
query15	0.94	0.88	0.88
query16	0.40	0.39	0.39
query17	1.07	1.09	1.06
query18	0.24	0.23	0.23
query19	2.02	1.86	1.87
query20	0.01	0.01	0.02
query21	15.40	1.02	0.70
query22	0.92	1.08	0.87
query23	14.69	1.53	0.76
query24	5.44	0.59	0.31
query25	0.18	0.10	0.09
query26	0.55	0.22	0.18
query27	0.09	0.09	0.09
query28	11.03	1.67	1.06
query29	12.53	4.09	3.35
query30	0.33	0.11	0.10
query31	2.83	0.62	0.43
query32	3.25	0.59	0.52
query33	3.12	3.06	3.13
query34	16.91	5.56	4.81
query35	4.84	4.85	4.93
query36	0.65	0.54	0.52
query37	0.22	0.19	0.20
query38	0.19	0.17	0.17
query39	0.06	0.05	0.05
query40	0.22	0.20	0.18
query41	0.12	0.06	0.06
query42	0.07	0.06	0.05
query43	0.06	0.06	0.05
Total cold run time: 105.46 s
Total hot run time: 31.17 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.53% (17751/33790)
Line Coverage 37.75% (161459/427754)
Region Coverage 32.18% (123122/382598)
Branch Coverage 33.62% (54104/160941)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.24% (23598/33125)
Line Coverage 57.70% (246679/427500)
Region Coverage 52.90% (205032/387548)
Branch Coverage 54.71% (88539/161830)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (10/10) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 16, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 16, 2025

PR approved by at least one committer and no changes requested.

@englefly englefly merged commit 456af82 into apache:master Oct 16, 2025
43 of 47 checks passed
@github-actions
Copy link
Contributor

github-actions bot commented Oct 16, 2025

PR approved by anyone and no changes requested.

@englefly englefly deleted the rf-thrift-fe-1015 branch October 16, 2025 06:59
github-actions bot pushed a commit that referenced this pull request Oct 16, 2025
@englefly
[opt](nereids)move runtime filter info from instance level to BE level (
529ab4a 
#56978)

### What problem does this PR solve?
Runtime filter info is shared among all instances within a BE.
This change saves thrift transfering effort.
@github-actions github-actions bot mentioned this pull request Oct 16, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 16, 2025
@github-actions @englefly
branch-4.0: [opt](nereids)move runtime filter info from instance leve…
85824c5 
…l to BE level #56978 (#57047)

Cherry-picked from #56978

Co-authored-by: minghong <zhouminghong@selectdb.com>
@englefly englefly mentioned this pull request Oct 20, 2025
Merged
16 tasks
BiteTheDDDDt pushed a commit that referenced this pull request Oct 21, 2025
@englefly
[fix](nereids)set RuntimeFilterInfo only on BE which is merge node (#…
33db05e 
…57108)

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information,
not runtime filter descriptor
In the previous pull request #56978, we moved the runtime info (which
records how runtime filters are merged) from the instance level to the
Backend (BE) level. In a multi-BE environment, when a runtime filter
needs to be merged, the BE does not perform the merge using the
mergeInstance specified by the Runtime Filter descriptor. Instead, it
selects a BE that has the RuntimeInfo set to perform the merge. This
causes BEs that should not be responsible for merging to wait for other
nodes to send local runtime filters, resulting in a time
github-actions bot pushed a commit that referenced this pull request Oct 21, 2025
@englefly
[fix](nereids)set RuntimeFilterInfo only on BE which is merge node (#…
f702d27 
…57108)

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information,
not runtime filter descriptor
In the previous pull request #56978, we moved the runtime info (which
records how runtime filters are merged) from the instance level to the
Backend (BE) level. In a multi-BE environment, when a runtime filter
needs to be merged, the BE does not perform the merge using the
mergeInstance specified by the Runtime Filter descriptor. Instead, it
selects a BE that has the RuntimeInfo set to perform the merge. This
causes BEs that should not be responsible for merging to wait for other
nodes to send local runtime filters, resulting in a time
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Oct 31, 2025
@englefly @BiteTheDDDDt
[fix](nereids)set RuntimeFilterInfo only on BE which is merge node (a…
334021f 
…pache#57108)

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information,
not runtime filter descriptor
In the previous pull request apache#56978, we moved the runtime info (which
records how runtime filters are merged) from the instance level to the
Backend (BE) level. In a multi-BE environment, when a runtime filter
needs to be merged, the BE does not perform the merge using the
mergeInstance specified by the Runtime Filter descriptor. Instead, it
selects a BE that has the RuntimeInfo set to perform the merge. This
causes BEs that should not be responsible for merging to wait for other
nodes to send local runtime filters, resulting in a time
@yiguolei yiguolei mentioned this pull request Nov 5, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@englefly @hello-stephen @doris-robot @BiteTheDDDDt @morrySnow @yiguolei