Skip to content

[fix](nereids)set RuntimeFilterInfo only on BE which is merge node #57108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 1 commit into from
Oct 21, 2025
Merged

[fix](nereids)set RuntimeFilterInfo only on BE which is merge node #57108

BiteTheDDDDt merged 1 commit into from
Oct 21, 2025

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Oct 17, 2025 •
edited
Loading

What problem does this PR solve?

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information, not runtime filter descriptor
In the previous pull request #56978, we moved the runtime info (which records how runtime filters are merged) from the instance level to the Backend (BE) level. In a multi-BE environment, when a runtime filter needs to be merged, the BE does not perform the merge using the mergeInstance specified by the Runtime Filter descriptor. Instead, it selects a BE that has the RuntimeInfo set to perform the merge. This causes BEs that should not be responsible for merging to wait for other nodes to send local runtime filters, resulting in a timeout.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@englefly
Copy link
Contributor Author

englefly commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 32.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae913b2bd6755b22dc1273b2c252c673745d10a4, data reload: false

query1	0.06	0.06	0.06
query2	0.11	0.06	0.06
query3	0.27	0.09	0.10
query4	1.62	0.13	0.12
query5	0.30	0.28	0.28
query6	1.18	0.68	0.68
query7	0.04	0.03	0.04
query8	0.06	0.06	0.05
query9	0.66	0.57	0.56
query10	0.64	0.63	0.63
query11	0.19	0.14	0.13
query12	0.19	0.15	0.14
query13	0.65	0.64	0.64
query14	1.09	1.05	1.07
query15	0.93	0.93	0.93
query16	0.43	0.46	0.45
query17	1.15	1.26	1.13
query18	0.24	0.23	0.22
query19	2.10	1.97	2.00
query20	0.02	0.02	0.02
query21	15.39	1.11	0.68
query22	0.79	1.26	0.71
query23	14.78	1.56	0.73
query24	7.23	0.80	0.97
query25	0.49	0.15	0.08
query26	0.64	0.19	0.17
query27	0.07	0.06	0.06
query28	9.09	1.49	0.96
query29	12.61	4.64	3.98
query30	0.29	0.15	0.13
query31	2.84	0.67	0.43
query32	3.25	0.59	0.50
query33	3.20	3.29	3.10
query34	16.27	5.52	4.88
query35	5.02	5.04	5.04
query36	0.72	0.58	0.52
query37	0.11	0.08	0.08
query38	0.07	0.06	0.05
query39	0.04	0.04	0.03
query40	0.20	0.16	0.16
query41	0.10	0.04	0.04
query42	0.05	0.03	0.04
query43	0.05	0.04	0.04
Total cold run time: 105.23 s
Total hot run time: 32.44 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/9) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

@englefly
Copy link
Contributor Author

englefly commented Oct 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.02% (29050/43346)
Region Coverage 67.30% (14369/21352)
Branch Coverage 57.66% (7637/13246)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/9) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

TPC-DS: Total hot run time: 190659 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fdd387aca088a76f66688ed8f3a44763e8338ab4, data reload: false

query1	1111	415	395	395
query2	6584	1708	1683	1683
query3	6763	223	227	223
query4	25898	23485	23189	23189
query5	4426	636	488	488
query6	328	240	246	240
query7	4655	515	303	303
query8	300	262	284	262
query9	8710	2688	2664	2664
query10	476	344	306	306
query11	15521	15092	15310	15092
query12	195	124	117	117
query13	1679	594	417	417
query14	11522	9212	9223	9212
query15	208	192	178	178
query16	7798	745	513	513
query17	1297	843	721	721
query18	2173	511	349	349
query19	245	239	195	195
query20	157	134	134	134
query21	220	146	133	133
query22	4861	4936	4525	4525
query23	34836	34401	33871	33871
query24	8392	2533	2529	2529
query25	673	538	466	466
query26	1296	290	167	167
query27	2843	555	395	395
query28	4823	2271	2220	2220
query29	817	666	523	523
query30	313	239	217	217
query31	952	848	786	786
query32	81	74	70	70
query33	591	421	342	342
query34	842	901	535	535
query35	842	881	810	810
query36	983	1052	951	951
query37	131	130	94	94
query38	3699	3665	3624	3624
query39	1484	1465	1432	1432
query40	224	131	124	124
query41	66	63	64	63
query42	124	127	114	114
query43	498	507	476	476
query44	1230	764	752	752
query45	189	188	174	174
query46	913	1007	647	647
query47	1795	1808	1760	1760
query48	414	440	325	325
query49	783	532	437	437
query50	666	691	419	419
query51	3915	4145	3890	3890
query52	113	117	100	100
query53	246	267	201	201
query54	634	621	537	537
query55	94	84	85	84
query56	416	340	324	324
query57	1171	1204	1129	1129
query58	289	282	286	282
query59	2691	2712	2497	2497
query60	358	356	334	334
query61	155	169	153	153
query62	782	737	654	654
query63	237	201	208	201
query64	4329	1206	894	894
query65	4071	4033	3961	3961
query66	1088	420	326	326
query67	15707	15331	15031	15031
query68	9782	953	602	602
query69	497	328	291	291
query70	1385	1283	1265	1265
query71	495	347	335	335
query72	5755	4948	4802	4802
query73	749	591	360	360
query74	8874	8923	8713	8713
query75	4498	3341	2772	2772
query76	5327	1164	742	742
query77	989	408	312	312
query78	9685	9695	8973	8973
query79	4489	798	572	572
query80	744	639	507	507
query81	498	280	230	230
query82	319	160	130	130
query83	316	270	252	252
query84	305	120	90	90
query85	860	470	427	427
query86	336	315	295	295
query87	3759	3796	3774	3774
query88	2887	2230	2217	2217
query89	438	329	298	298
query90	2178	218	217	217
query91	162	170	134	134
query92	86	72	70	70
query93	2845	984	637	637
query94	691	436	335	335
query95	411	323	320	320
query96	508	579	293	293
query97	2937	2973	2918	2918
query98	238	228	209	209
query99	1422	1451	1307	1307
Total cold run time: 286388 ms
Total hot run time: 190659 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

ClickBench: Total hot run time: 28.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fdd387aca088a76f66688ed8f3a44763e8338ab4, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.04	0.06
query3	0.26	0.08	0.08
query4	1.61	0.12	0.12
query5	0.30	0.26	0.25
query6	1.20	0.67	0.66
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.63	0.53	0.52
query10	0.58	0.59	0.59
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.62	0.62
query14	1.03	1.04	1.04
query15	0.87	0.89	0.86
query16	0.40	0.40	0.41
query17	1.06	1.12	1.04
query18	0.25	0.20	0.21
query19	1.97	1.87	1.82
query20	0.02	0.02	0.02
query21	15.45	0.19	0.12
query22	5.18	0.07	0.05
query23	15.69	0.25	0.10
query24	2.58	0.50	0.65
query25	0.08	0.08	0.06
query26	0.14	0.13	0.15
query27	0.06	0.05	0.07
query28	4.75	1.16	0.95
query29	12.56	4.04	3.27
query30	0.30	0.15	0.13
query31	2.83	0.61	0.39
query32	3.24	0.57	0.49
query33	3.10	3.12	3.07
query34	16.10	5.50	4.89
query35	4.91	4.95	4.87
query36	0.71	0.51	0.51
query37	0.11	0.07	0.07
query38	0.07	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 99.6 s
Total hot run time: 28.53 s

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.64% (17895/33995)
Line Coverage 37.86% (162309/428747)
Region Coverage 32.26% (123729/383544)
Branch Coverage 33.67% (54222/161025)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.40% (23796/33328)
Line Coverage 57.78% (247575/428499)
Region Coverage 52.83% (205228/388502)
Branch Coverage 54.56% (88345/161920)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2025

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 33db05e into apache:master Oct 21, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 21, 2025
@englefly
[fix](nereids)set RuntimeFilterInfo only on BE which is merge node (#…
f702d27 
…57108)

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information,
not runtime filter descriptor
In the previous pull request #56978, we moved the runtime info (which
records how runtime filters are merged) from the instance level to the
Backend (BE) level. In a multi-BE environment, when a runtime filter
needs to be merged, the BE does not perform the merge using the
mergeInstance specified by the Runtime Filter descriptor. Instead, it
selects a BE that has the RuntimeInfo set to perform the merge. This
causes BEs that should not be responsible for merging to wait for other
nodes to send local runtime filters, resulting in a time
@github-actions github-actions bot mentioned this pull request Oct 21, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 23, 2025
@github-actions @englefly
branch-4.0: [fix](nereids)set RuntimeFilterInfo only on BE which is m…
e05ceb5 
…erge node #57108 (#57199)

Cherry-picked from #57108

Co-authored-by: minghong <zhouminghong@selectdb.com>
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Oct 31, 2025
@englefly @BiteTheDDDDt
[fix](nereids)set RuntimeFilterInfo only on BE which is merge node (a…
334021f 
…pache#57108)

set RuntimeFilterInfo only on BE which is merge node.
note: RuntimeFilterInfo only contains runtime-filter merge information,
not runtime filter descriptor
In the previous pull request apache#56978, we moved the runtime info (which
records how runtime filters are merged) from the instance level to the
Backend (BE) level. In a multi-BE environment, when a runtime filter
needs to be merged, the BE does not perform the merge using the
mergeInstance specified by the Runtime Filter descriptor. Instead, it
selects a BE that has the RuntimeInfo set to perform the merge. This
causes BEs that should not be responsible for merging to wait for other
nodes to send local runtime filters, resulting in a time
@yiguolei yiguolei mentioned this pull request Nov 5, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@englefly @hello-stephen @doris-robot @BiteTheDDDDt @starocean999 @yiguolei