Skip to content

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Sep 17, 2025

What problem does this PR solve?

Optimize Variant sparse-column scans by introducing a per-segment sparse column cache so multiple readers/iterators reuse sparse column (keyed by column unique_id and path).

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

run buildall

1 similar comment
@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34850 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3418f1dbf408ba45fefc1aa043c6bf8988bcc02b, data reload: false

------ Round 1 ----------------------------------
q1	17630	5204	5037	5037
q2	2043	330	222	222
q3	10225	1328	729	729
q4	10234	1026	545	545
q5	7541	2439	2412	2412
q6	187	171	139	139
q7	972	767	645	645
q8	9354	1403	1159	1159
q9	7038	5161	5204	5161
q10	6894	2404	1957	1957
q11	495	298	282	282
q12	354	366	230	230
q13	17759	3657	3027	3027
q14	260	242	229	229
q15	552	489	500	489
q16	1012	1006	941	941
q17	603	904	353	353
q18	7571	7174	7151	7151
q19	1254	981	567	567
q20	345	359	221	221
q21	3926	3284	2358	2358
q22	1065	1031	996	996
Total cold run time: 107314 ms
Total hot run time: 34850 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5098	5080	5082	5080
q2	264	340	224	224
q3	2245	2682	2336	2336
q4	1368	1760	1374	1374
q5	4182	4566	4602	4566
q6	221	174	133	133
q7	2097	1977	1803	1803
q8	2652	2697	2645	2645
q9	7516	7516	7271	7271
q10	3163	3360	2863	2863
q11	621	519	518	518
q12	712	820	670	670
q13	3504	3963	3390	3390
q14	307	327	279	279
q15	521	488	516	488
q16	1076	1155	1100	1100
q17	1201	1540	1387	1387
q18	7931	7656	7571	7571
q19	837	804	861	804
q20	2003	2103	1905	1905
q21	5163	4405	4280	4280
q22	1089	1032	998	998
Total cold run time: 53771 ms
Total hot run time: 51685 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189228 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3418f1dbf408ba45fefc1aa043c6bf8988bcc02b, data reload: false

query1	1088	442	408	408
query2	6565	1728	1695	1695
query3	6749	233	234	233
query4	25903	23527	23274	23274
query5	4852	617	474	474
query6	333	237	222	222
query7	4678	530	302	302
query8	295	262	242	242
query9	8678	2642	2664	2642
query10	498	349	263	263
query11	15828	14965	15175	14965
query12	187	121	114	114
query13	1679	559	447	447
query14	11918	9243	9358	9243
query15	208	194	173	173
query16	7664	661	485	485
query17	1356	736	624	624
query18	2053	446	357	357
query19	205	195	166	166
query20	129	129	127	127
query21	220	132	124	124
query22	4180	4335	4136	4136
query23	33844	32955	32912	32912
query24	8396	2376	2413	2376
query25	593	550	476	476
query26	1251	281	166	166
query27	2733	514	373	373
query28	4393	2263	2249	2249
query29	820	618	507	507
query30	286	230	208	208
query31	918	829	732	732
query32	87	76	74	74
query33	593	417	343	343
query34	801	861	533	533
query35	801	853	750	750
query36	1061	1017	914	914
query37	117	106	84	84
query38	3498	3524	3466	3466
query39	1492	1432	1415	1415
query40	221	131	118	118
query41	67	62	61	61
query42	129	116	111	111
query43	511	509	484	484
query44	1351	865	859	859
query45	183	178	171	171
query46	879	1028	647	647
query47	1771	1810	1768	1768
query48	394	421	322	322
query49	768	534	425	425
query50	645	690	402	402
query51	3894	3948	3959	3948
query52	122	112	103	103
query53	235	260	204	204
query54	596	592	523	523
query55	95	83	87	83
query56	312	325	309	309
query57	1216	1225	1114	1114
query58	272	268	267	267
query59	2581	2651	2591	2591
query60	347	348	337	337
query61	185	163	168	163
query62	804	768	678	678
query63	232	199	190	190
query64	4498	1151	868	868
query65	4047	3949	3987	3949
query66	1117	447	342	342
query67	15758	15683	15443	15443
query68	8997	939	591	591
query69	491	323	290	290
query70	1296	1273	1278	1273
query71	578	341	309	309
query72	6052	5028	5007	5007
query73	749	628	370	370
query74	8864	9186	8662	8662
query75	4212	3338	2792	2792
query76	3709	1200	748	748
query77	793	397	311	311
query78	9626	9777	8888	8888
query79	2064	883	593	593
query80	640	564	507	507
query81	519	278	297	278
query82	490	169	130	130
query83	279	263	249	249
query84	267	110	90	90
query85	889	470	426	426
query86	396	316	305	305
query87	3743	3746	3643	3643
query88	3664	2256	2256	2256
query89	403	343	297	297
query90	1863	220	216	216
query91	169	168	149	149
query92	90	69	66	66
query93	1686	1026	646	646
query94	697	455	340	340
query95	415	318	306	306
query96	483	585	288	288
query97	2944	3008	2904	2904
query98	249	215	213	213
query99	1648	1430	1311	1311
Total cold run time: 279145 ms
Total hot run time: 189228 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3418f1dbf408ba45fefc1aa043c6bf8988bcc02b, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.28	0.26	0.25
query6	1.18	0.67	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.63	0.54	0.52
query10	0.58	0.59	0.57
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.66	0.63	0.61
query14	1.04	1.02	1.05
query15	0.87	0.88	0.87
query16	0.40	0.41	0.39
query17	1.07	1.04	1.04
query18	0.21	0.20	0.20
query19	1.91	1.91	1.90
query20	0.02	0.02	0.02
query21	15.44	0.94	0.58
query22	0.74	1.20	0.81
query23	14.79	1.43	0.67
query24	7.00	1.15	0.33
query25	0.39	0.30	0.09
query26	0.60	0.16	0.14
query27	0.07	0.06	0.05
query28	9.51	0.94	0.43
query29	12.58	3.94	3.27
query30	0.28	0.14	0.11
query31	2.83	0.62	0.38
query32	3.23	0.55	0.48
query33	3.09	3.12	3.18
query34	16.25	5.50	4.85
query35	4.93	4.94	4.95
query36	0.68	0.51	0.50
query37	0.10	0.07	0.08
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.23 s
Total hot run time: 29.77 s

@csun5285
Copy link
Contributor Author

run buildall

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1496 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 510f050de0b3b34d188305b295b61dab29bf89eb, data reload: false

------ Round 1 ----------------------------------
q1	17621	29	23	23
q2	2639	26	25	25
q3	10722	15	14	14
q4	11157	16	15	15
q5	8103	15	15	15
q6	230	15	14	14
q7	939	26	22	22
q8	9439	15	12	12
q9	7363	13	12	12
q10	6971	13	13	13
q11	649	22	20	20
q12	342	13	11	11
q13	18684	13	11	11
q14	262	12	12	12
q15	569	13	12	12
q16	1046	999	956	956
q17	560	13	12	12
q18	8142	12	10	10
q19	1167	13	12	12
q20	1998	371	242	242
q21	9090	22	21	21
q22	1104	14	12	12
Total cold run time: 118797 ms
Total hot run time: 1496 ms

----- Round 2, with runtime_filter_mode=off -----
q1	13	13	12	12
q2	22	22	21	21
q3	12	12	13	12
q4	14	12	14	12
q5	12	12	12	12
q6	11	12	11	11
q7	19	19	19	19
q8	10	12	11	11
q9	10	11	11	11
q10	10	10	10	10
q11	19	21	20	20
q12	11	10	10	10
q13	10	10	10	10
q14	11	10	11	10
q15	11	11	10	10
q16	1065	1106	1037	1037
q17	12	11	10	10
q18	10	9	10	9
q19	10	10	10	10
q20	1910	1994	1861	1861
q21	21	20	20	20
q22	10	10	10	10
Total cold run time: 3233 ms
Total hot run time: 3148 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2756 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 510f050de0b3b34d188305b295b61dab29bf89eb, data reload: false

query1	1089	20	12	12
query2	7115	18	16	16
query3	7627	11	11	11
query4	26894	12	12	12
query5	4473	12	11	11
query6	367	12	12	12
query7	5442	12	11	11
query8	354	18	17	17
query9	9228	11	10	10
query10	727	12	10	10
query11	16685	10	9	9
query12	177	11	10	10
query13	1715	10	8	8
query14	10739	15	12	12
query15	228	11	10	10
query16	7179	11	10	10
query17	1485	12	36	12
query18	2824	12	10	10
query19	203	10	8	8
query20	136	10	9	9
query21	223	10	11	10
query22	4166	11	9	9
query23	33817	19	12	12
query24	9802	14	11	11
query25	749	10	10	10
query26	1666	9	10	9
query27	3377	10	9	9
query28	6384	12	8	8
query29	1684	10	7	7
query30	639	13	10	10
query31	1717	11	10	10
query32	121	12	9	9
query33	1335	11	9	9
query34	1682	844	535	535
query35	935	11	11	11
query36	998	11	11	11
query37	122	12	10	10
query38	3564	10	9	9
query39	1490	729	732	729
query40	320	12	10	10
query41	83	13	11	11
query42	146	10	9	9
query43	508	10	11	10
query44	1342	10	10	10
query45	197	11	10	10
query46	916	11	11	11
query47	1830	13	8	8
query48	394	10	9	9
query49	1219	11	9	9
query50	755	11	9	9
query51	4051	9	9	9
query52	127	9	8	8
query53	231	10	9	9
query54	705	11	10	10
query55	89	9	8	8
query56	329	10	12	10
query57	1247	10	9	9
query58	306	12	10	10
query59	2651	9	9	9
query60	369	10	9	9
query61	175	9	8	8
query62	923	10	9	9
query63	262	10	10	10
query64	5502	11	9	9
query65	4013	9	9	9
query66	1628	11	11	11
query67	16947	19	8	8
query68	4193	10	9	9
query69	892	10	8	8
query70	1398	9	8	8
query71	439	354	342	342
query72	7159	11	11	11
query73	533	9	9	9
query74	9494	11	10	10
query75	3357	10	11	10
query76	2530	11	11	11
query77	969	10	10	10
query78	9863	15	11	11
query79	1118	11	9	9
query80	695	9	12	9
query81	530	10	9	9
query82	204	9	8	8
query83	300	10	10	10
query84	405	9	8	8
query85	1015	12	9	9
query86	458	9	9	9
query87	3927	10	9	9
query88	2814	10	9	9
query89	377	9	9	9
query90	2184	10	9	9
query91	184	10	10	10
query92	88	10	11	10
query93	1094	11	13	11
query94	1389	10	10	10
query95	528	11	10	10
query96	445	10	10	10
query97	3103	11	11	11
query98	293	258	220	220
query99	1471	10	9	9
Total cold run time: 289872 ms
Total hot run time: 2756 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 510f050de0b3b34d188305b295b61dab29bf89eb, data reload: false

query1	0.07	0.02	0.00
query2	0.10	0.01	0.01
query3	0.27	0.01	0.00
query4	1.76	0.00	0.01
query5	0.28	0.01	0.00
query6	1.66	0.01	0.01
query7	0.05	0.01	0.00
query8	0.06	0.00	0.01
query9	0.62	0.00	0.00
query10	0.59	0.01	0.00
query11	0.17	0.00	0.00
query12	0.17	0.00	0.00
query13	0.64	0.00	0.00
query14	1.08	0.00	0.01
query15	0.89	0.00	0.00
query16	0.41	0.01	0.00
query17	1.05	0.01	0.01
query18	0.22	0.00	0.00
query19	1.91	0.00	0.00
query20	0.02	0.00	0.00
query21	15.36	0.00	0.00
query22	5.79	0.00	0.00
query23	15.74	0.00	0.01
query24	1.27	0.00	0.00
query25	0.22	0.00	0.00
query26	0.16	0.00	0.00
query27	0.13	0.00	0.00
query28	1.17	0.00	0.00
query29	13.34	0.00	0.00
query30	0.30	0.00	0.01
query31	2.41	0.00	0.01
query32	5.91	0.00	0.00
query33	4.34	0.01	0.01
query34	7.35	0.00	0.00
query35	6.33	0.00	0.01
query36	0.70	0.00	0.01
query37	0.11	0.01	0.01
query38	0.08	0.00	0.01
query39	0.05	0.01	0.00
query40	0.18	0.00	0.00
query41	0.10	0.01	0.00
query42	0.06	0.01	0.01
query43	0.05	0.00	0.00
Total cold run time: 93.17 s
Total hot run time: 0.06 s

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1517 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a881056bae3f63cdb6a16a19ca7e0bb6c554ee53, data reload: false

------ Round 1 ----------------------------------
q1	17654	27	19	19
q2	2614	29	27	27
q3	10711	15	15	15
q4	11153	16	15	15
q5	8120	17	15	15
q6	239	17	14	14
q7	950	24	24	24
q8	9488	15	13	13
q9	7397	14	14	14
q10	6969	14	12	12
q11	715	21	20	20
q12	361	13	12	12
q13	18608	14	12	12
q14	253	13	13	13
q15	617	14	12	12
q16	1081	974	982	974
q17	582	14	12	12
q18	7691	12	11	11
q19	1400	13	12	12
q20	1953	374	240	240
q21	9246	20	22	20
q22	1130	12	11	11
Total cold run time: 118932 ms
Total hot run time: 1517 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	11	11	11
q2	20	20	19	19
q3	12	10	11	10
q4	12	11	11	11
q5	10	10	13	10
q6	11	10	10	10
q7	19	19	19	19
q8	11	10	10	10
q9	10	10	10	10
q10	10	10	10	10
q11	20	20	18	18
q12	11	11	10	10
q13	11	10	10	10
q14	11	10	12	10
q15	11	10	10	10
q16	1042	1090	1039	1039
q17	12	11	10	10
q18	10	10	11	10
q19	10	13	10	10
q20	1860	2038	1835	1835
q21	20	20	19	19
q22	10	10	9	9
Total cold run time: 3154 ms
Total hot run time: 3110 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2799 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a881056bae3f63cdb6a16a19ca7e0bb6c554ee53, data reload: false

query1	1075	20	11	11
query2	7130	17	14	14
query3	7640	12	11	11
query4	26339	11	10	10
query5	4480	13	12	12
query6	370	12	12	12
query7	5442	12	12	12
query8	361	19	18	18
query9	9210	13	11	11
query10	737	12	10	10
query11	16072	11	9	9
query12	174	12	9	9
query13	1723	11	9	9
query14	10921	17	13	13
query15	249	13	11	11
query16	7184	11	10	10
query17	1516	10	11	10
query18	2818	11	10	10
query19	206	11	10	10
query20	140	11	10	10
query21	224	10	9	9
query22	4017	10	9	9
query23	33828	16	13	13
query24	9577	14	11	11
query25	715	11	9	9
query26	1236	11	11	11
query27	3426	10	8	8
query28	6364	10	10	10
query29	1232	11	9	9
query30	612	12	10	10
query31	1739	13	10	10
query32	115	11	10	10
query33	1334	10	9	9
query34	1711	812	522	522
query35	902	10	10	10
query36	1017	10	10	10
query37	132	10	9	9
query38	3544	9	8	8
query39	1498	750	743	743
query40	230	11	12	11
query41	83	12	11	11
query42	156	10	11	10
query43	502	10	9	9
query44	1383	10	10	10
query45	202	11	11	11
query46	938	10	9	9
query47	1784	9	9	9
query48	413	11	10	10
query49	1175	11	10	10
query50	775	11	10	10
query51	4008	10	9	9
query52	114	10	9	9
query53	249	10	10	10
query54	723	10	10	10
query55	95	11	10	10
query56	335	11	10	10
query57	1257	11	9	9
query58	324	10	11	10
query59	2645	10	9	9
query60	402	10	10	10
query61	211	11	10	10
query62	835	10	10	10
query63	271	12	11	11
query64	4593	11	12	11
query65	4041	10	8	8
query66	1492	11	11	11
query67	16539	26	11	11
query68	3491	12	9	9
query69	556	11	11	11
query70	1362	10	10	10
query71	427	364	357	357
query72	7032	11	9	9
query73	720	13	11	11
query74	9412	11	11	11
query75	3433	10	12	10
query76	2656	10	52	10
query77	910	10	11	10
query78	9881	15	9	9
query79	1498	10	11	10
query80	917	11	10	10
query81	1191	10	9	9
query82	231	10	10	10
query83	296	10	12	10
query84	334	9	9	9
query85	1639	10	11	10
query86	839	9	8	8
query87	3921	10	9	9
query88	2992	11	11	11
query89	387	10	8	8
query90	2174	10	9	9
query91	176	10	10	10
query92	100	10	8	8
query93	1917	10	8	8
query94	1655	9	9	9
query95	544	11	9	9
query96	414	10	8	8
query97	3087	9	8	8
query98	233	236	230	230
query99	1442	9	9	9
Total cold run time: 288652 ms
Total hot run time: 2799 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a881056bae3f63cdb6a16a19ca7e0bb6c554ee53, data reload: false

query1	0.07	0.01	0.00
query2	0.10	0.01	0.01
query3	0.26	0.00	0.00
query4	1.74	0.00	0.01
query5	0.29	0.01	0.00
query6	1.67	0.01	0.00
query7	0.05	0.01	0.01
query8	0.07	0.00	0.00
query9	0.62	0.00	0.01
query10	0.59	0.00	0.00
query11	0.18	0.00	0.00
query12	0.16	0.00	0.01
query13	0.64	0.01	0.01
query14	1.06	0.00	0.00
query15	0.89	0.00	0.01
query16	0.43	0.00	0.00
query17	1.04	0.01	0.00
query18	0.22	0.01	0.01
query19	1.93	0.01	0.01
query20	0.02	0.00	0.01
query21	15.36	0.01	0.01
query22	6.02	0.00	0.00
query23	16.50	0.01	0.00
query24	1.30	0.00	0.00
query25	0.23	0.00	0.00
query26	0.17	0.00	0.01
query27	0.11	0.01	0.00
query28	1.16	0.00	0.01
query29	13.22	0.00	0.00
query30	0.31	0.00	0.00
query31	2.25	0.00	0.00
query32	5.83	0.00	0.00
query33	4.36	0.01	0.00
query34	7.42	0.00	0.00
query35	6.25	0.00	0.00
query36	0.69	0.00	0.00
query37	0.11	0.01	0.01
query38	0.08	0.00	0.00
query39	0.05	0.00	0.00
query40	0.19	0.00	0.00
query41	0.10	0.00	0.00
query42	0.07	0.01	0.00
query43	0.05	0.01	0.00
Total cold run time: 93.86 s
Total hot run time: 0.07 s

@csun5285
Copy link
Contributor Author

run buildall

1 similar comment
@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1543 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c59ec9ea826fc52ecec74be07594cfaf30c747c7, data reload: false

------ Round 1 ----------------------------------
q1	6657	28	20	20
q2	744	26	29	26
q3	885	17	15	15
q4	5591	16	16	16
q5	2286	17	16	16
q6	231	14	14	14
q7	950	23	21	21
q8	1212	15	13	13
q9	7148	24	12	12
q10	6667	13	12	12
q11	466	25	22	22
q12	375	14	12	12
q13	17697	13	12	12
q14	254	12	12	12
q15	631	11	10	10
q16	1069	998	987	987
q17	582	12	10	10
q18	7906	12	11	11
q19	1120	12	12	12
q20	1287	382	257	257
q21	6755	22	22	22
q22	1110	13	11	11
Total cold run time: 71623 ms
Total hot run time: 1543 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	11	10	10
q2	19	20	20	20
q3	11	11	11	11
q4	13	12	11	11
q5	11	10	10	10
q6	10	11	11	11
q7	20	19	19	19
q8	11	11	11	11
q9	11	11	11	11
q10	11	10	11	10
q11	18	19	20	19
q12	10	10	11	10
q13	10	10	11	10
q14	12	11	12	11
q15	10	10	10	10
q16	1086	1083	1022	1022
q17	13	10	10	10
q18	11	10	10	10
q19	11	11	11	11
q20	1880	1988	1826	1826
q21	23	22	20	20
q22	12	10	10	10
Total cold run time: 3224 ms
Total hot run time: 3093 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2720 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c59ec9ea826fc52ecec74be07594cfaf30c747c7, data reload: false

query1	1066	21	13	13
query2	7145	19	15	15
query3	7629	11	10	10
query4	26498	11	11	11
query5	4440	13	12	12
query6	366	11	10	10
query7	5454	10	10	10
query8	353	22	16	16
query9	9245	11	11	11
query10	732	13	12	12
query11	16276	12	9	9
query12	175	12	10	10
query13	1717	11	11	11
query14	10874	14	13	13
query15	228	12	10	10
query16	7383	10	10	10
query17	2137	12	10	10
query18	2678	12	10	10
query19	220	10	9	9
query20	141	11	10	10
query21	225	11	10	10
query22	4130	11	10	10
query23	34002	36	18	18
query24	10645	13	11	11
query25	735	11	9	9
query26	1829	10	9	9
query27	3364	11	9	9
query28	6328	12	11	11
query29	1870	10	10	10
query30	625	12	11	11
query31	1741	10	9	9
query32	119	11	9	9
query33	1351	10	8	8
query34	1640	822	531	531
query35	888	9	10	9
query36	988	9	9	9
query37	253	9	8	8
query38	3569	10	9	9
query39	1522	762	745	745
query40	310	10	9	9
query41	77	12	11	11
query42	148	12	9	9
query43	500	11	9	9
query44	1351	10	9	9
query45	189	9	9	9
query46	904	10	9	9
query47	1836	10	9	9
query48	408	9	9	9
query49	1294	10	9	9
query50	741	9	8	8
query51	4069	9	9	9
query52	125	10	9	9
query53	240	11	9	9
query54	744	8	8	8
query55	95	10	9	9
query56	350	10	9	9
query57	1245	10	10	10
query58	302	13	11	11
query59	2636	9	8	8
query60	375	9	9	9
query61	177	10	9	9
query62	835	9	7	7
query63	267	12	11	11
query64	5520	10	9	9
query65	4059	9	8	8
query66	1606	10	9	9
query67	15306	23	8	8
query68	3401	9	9	9
query69	532	9	8	8
query70	1384	9	9	9
query71	413	349	327	327
query72	6462	8	8	8
query73	512	10	8	8
query74	9477	10	10	10
query75	3332	9	9	9
query76	3033	10	8	8
query77	702	9	9	9
query78	9392	11	9	9
query79	1016	9	9	9
query80	626	10	10	10
query81	489	10	8	8
query82	208	9	9	9
query83	278	10	9	9
query84	271	9	8	8
query85	917	9	8	8
query86	360	10	9	9
query87	3753	9	9	9
query88	2794	11	9	9
query89	372	9	10	9
query90	2152	9	8	8
query91	174	8	8	8
query92	92	10	10	10
query93	1062	9	8	8
query94	771	9	9	9
query95	504	10	8	8
query96	418	10	8	8
query97	2939	9	8	8
query98	227	234	213	213
query99	1457	10	10	10
Total cold run time: 285810 ms
Total hot run time: 2720 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c59ec9ea826fc52ecec74be07594cfaf30c747c7, data reload: false

query1	0.07	0.02	0.01
query2	0.10	0.01	0.00
query3	0.26	0.01	0.01
query4	1.75	0.01	0.00
query5	0.29	0.01	0.01
query6	1.67	0.00	0.01
query7	0.06	0.01	0.00
query8	0.06	0.00	0.00
query9	0.64	0.00	0.01
query10	0.62	0.00	0.00
query11	0.18	0.01	0.00
query12	0.16	0.00	0.00
query13	0.74	0.00	0.00
query14	1.05	0.00	0.01
query15	0.91	0.00	0.00
query16	0.42	0.00	0.00
query17	1.12	0.01	0.01
query18	0.22	0.00	0.00
query19	1.94	0.00	0.00
query20	0.02	0.00	0.00
query21	15.36	0.00	0.01
query22	5.90	0.00	0.00
query23	15.76	0.01	0.01
query24	1.51	0.00	0.00
query25	0.22	0.00	0.00
query26	0.17	0.01	0.01
query27	0.13	0.00	0.00
query28	1.22	0.00	0.00
query29	13.28	0.00	0.01
query30	0.31	0.00	0.00
query31	2.29	0.00	0.00
query32	5.95	0.00	0.00
query33	4.37	0.00	0.00
query34	7.41	0.00	0.00
query35	6.15	0.01	0.01
query36	0.67	0.00	0.00
query37	0.11	0.00	0.00
query38	0.08	0.00	0.00
query39	0.05	0.01	0.00
query40	0.20	0.00	0.00
query41	0.09	0.00	0.00
query42	0.06	0.00	0.00
query43	0.05	0.00	0.00
Total cold run time: 93.62 s
Total hot run time: 0.07 s

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1529 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e8cdfb3faa7b4e5545c535e282a85af6d470c51, data reload: false

------ Round 1 ----------------------------------
q1	17618	36	18	18
q2	2645	26	25	25
q3	10730	19	15	15
q4	11206	17	18	17
q5	8113	25	17	17
q6	235	13	13	13
q7	963	26	23	23
q8	9813	13	13	13
q9	9874	15	12	12
q10	7022	13	12	12
q11	713	22	20	20
q12	856	12	11	11
q13	18688	13	12	12
q14	253	11	11	11
q15	632	11	10	10
q16	1083	991	958	958
q17	587	13	11	11
q18	8073	13	11	11
q19	1265	13	11	11
q20	1946	379	277	277
q21	9148	20	20	20
q22	1085	13	12	12
Total cold run time: 122548 ms
Total hot run time: 1529 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	11	12	11
q2	22	20	20	20
q3	12	11	11	11
q4	15	11	11	11
q5	12	12	12	12
q6	12	12	11	11
q7	21	19	19	19
q8	11	10	11	10
q9	10	10	10	10
q10	11	10	10	10
q11	19	24	18	18
q12	11	12	10	10
q13	10	10	10	10
q14	14	12	14	12
q15	11	10	10	10
q16	1051	1085	1033	1033
q17	12	11	10	10
q18	10	10	11	10
q19	11	10	11	10
q20	1872	1984	1843	1843
q21	21	20	18	18
q22	12	11	11	11
Total cold run time: 3192 ms
Total hot run time: 3120 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2818 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e8cdfb3faa7b4e5545c535e282a85af6d470c51, data reload: false

query1	1095	20	11	11
query2	7159	18	15	15
query3	7633	12	11	11
query4	26898	12	12	12
query5	4506	12	12	12
query6	365	10	9	9
query7	5461	11	10	10
query8	363	17	18	17
query9	9229	11	10	10
query10	744	13	11	11
query11	15740	12	11	11
query12	176	13	11	11
query13	1724	13	11	11
query14	10880	15	14	14
query15	227	11	11	11
query16	7177	10	9	9
query17	1535	11	10	10
query18	2820	12	9	9
query19	215	11	10	10
query20	132	11	10	10
query21	217	12	11	11
query22	3997	10	11	10
query23	33561	19	14	14
query24	9410	13	15	13
query25	716	11	10	10
query26	1075	11	10	10
query27	3369	10	10	10
query28	6350	10	10	10
query29	1190	10	10	10
query30	612	12	10	10
query31	1749	11	11	11
query32	121	13	11	11
query33	1184	11	9	9
query34	1587	813	518	518
query35	935	10	10	10
query36	1005	9	8	8
query37	122	10	8	8
query38	3610	10	9	9
query39	1508	758	779	758
query40	232	11	9	9
query41	88	11	11	11
query42	156	9	10	9
query43	508	9	9	9
query44	1349	11	8	8
query45	192	11	9	9
query46	914	9	8	8
query47	1787	10	9	9
query48	420	10	10	10
query49	1125	10	9	9
query50	773	9	9	9
query51	4069	10	10	10
query52	125	9	8	8
query53	242	11	10	10
query54	730	10	9	9
query55	95	10	8	8
query56	334	10	10	10
query57	1223	9	9	9
query58	329	11	10	10
query59	2679	10	8	8
query60	364	10	10	10
query61	174	10	10	10
query62	879	10	11	10
query63	262	10	10	10
query64	4085	12	11	11
query65	4026	10	9	9
query66	1081	10	9	9
query67	16386	35	13	13
query68	3860	12	10	10
query69	548	9	8	8
query70	1466	11	11	11
query71	416	347	351	347
query72	7141	13	12	12
query73	535	12	9	9
query74	9443	11	10	10
query75	3448	12	12	12
query76	2669	10	10	10
query77	792	11	10	10
query78	9979	16	11	11
query79	1241	11	10	10
query80	1259	11	11	11
query81	594	11	10	10
query82	1476	11	9	9
query83	406	11	11	11
query84	316	10	11	10
query85	1497	11	14	11
query86	724	9	9	9
query87	4047	9	9	9
query88	2893	17	12	12
query89	397	9	8	8
query90	3386	10	10	10
query91	174	10	9	9
query92	79	11	11	11
query93	1671	10	11	10
query94	1471	11	9	9
query95	518	12	11	11
query96	425	9	8	8
query97	3115	9	8	8
query98	269	234	238	234
query99	1452	10	12	10
Total cold run time: 288731 ms
Total hot run time: 2818 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e8cdfb3faa7b4e5545c535e282a85af6d470c51, data reload: false

query1	0.06	0.02	0.01
query2	0.10	0.00	0.01
query3	0.27	0.01	0.01
query4	1.77	0.01	0.01
query5	0.29	0.01	0.00
query6	1.65	0.01	0.00
query7	0.04	0.00	0.00
query8	0.06	0.00	0.01
query9	0.63	0.00	0.00
query10	0.60	0.00	0.00
query11	0.17	0.01	0.01
query12	0.16	0.01	0.00
query13	0.64	0.01	0.01
query14	1.07	0.00	0.00
query15	0.88	0.00	0.00
query16	0.40	0.00	0.00
query17	1.05	0.00	0.00
query18	0.22	0.01	0.00
query19	2.02	0.00	0.00
query20	0.02	0.00	0.00
query21	15.37	0.00	0.01
query22	6.10	0.00	0.01
query23	16.55	0.00	0.01
query24	1.40	0.01	0.00
query25	0.20	0.00	0.01
query26	0.17	0.00	0.00
query27	0.11	0.00	0.00
query28	1.10	0.00	0.00
query29	13.31	0.00	0.01
query30	0.30	0.00	0.00
query31	2.27	0.00	0.01
query32	5.91	0.00	0.00
query33	4.34	0.01	0.01
query34	7.18	0.00	0.00
query35	5.76	0.01	0.01
query36	0.70	0.01	0.00
query37	0.11	0.01	0.01
query38	0.07	0.00	0.01
query39	0.05	0.00	0.00
query40	0.18	0.01	0.00
query41	0.10	0.00	0.00
query42	0.06	0.01	0.01
query43	0.05	0.00	0.00
Total cold run time: 93.49 s
Total hot run time: 0.09 s

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1486 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94b85d805b3d4fbb68391cfb062748f17212d106, data reload: false

------ Round 1 ----------------------------------
q1	17626	28	20	20
q2	2629	28	26	26
q3	10711	15	14	14
q4	11158	15	17	15
q5	8104	15	16	15
q6	231	14	13	13
q7	967	24	22	22
q8	9462	14	12	12
q9	12937	14	12	12
q10	6967	12	12	12
q11	733	22	21	21
q12	349	12	11	11
q13	18581	12	13	12
q14	249	13	11	11
q15	623	12	10	10
q16	1059	979	953	953
q17	590	12	11	11
q18	7823	13	12	12
q19	1142	12	11	11
q20	1138	369	241	241
q21	9105	21	21	21
q22	1077	12	11	11
Total cold run time: 123261 ms
Total hot run time: 1486 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	11	11	11
q2	21	20	20	20
q3	12	11	11	11
q4	13	12	12	12
q5	11	10	10	10
q6	10	10	10	10
q7	20	19	18	18
q8	10	10	10	10
q9	10	11	10	10
q10	10	35	11	11
q11	20	21	19	19
q12	10	10	11	10
q13	9	10	10	10
q14	10	12	13	12
q15	10	9	9	9
q16	1084	1093	1032	1032
q17	11	10	9	9
q18	10	10	10	10
q19	10	10	9	9
q20	1863	1950	1892	1892
q21	20	18	19	18
q22	10	10	9	9
Total cold run time: 3196 ms
Total hot run time: 3162 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2756 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94b85d805b3d4fbb68391cfb062748f17212d106, data reload: false

query1	1064	19	11	11
query2	7122	18	15	15
query3	7634	11	10	10
query4	26460	12	12	12
query5	4503	13	11	11
query6	391	11	11	11
query7	5424	11	11	11
query8	368	19	19	19
query9	9195	13	11	11
query10	720	12	12	12
query11	16056	12	11	11
query12	190	11	10	10
query13	1744	12	10	10
query14	10935	14	12	12
query15	403	11	9	9
query16	7205	11	10	10
query17	1540	11	11	11
query18	3075	13	11	11
query19	242	11	9	9
query20	137	11	10	10
query21	235	11	9	9
query22	4172	10	10	10
query23	33670	20	12	12
query24	9806	12	11	11
query25	735	11	9	9
query26	1615	11	10	10
query27	3484	9	8	8
query28	6312	10	10	10
query29	1421	10	9	9
query30	607	11	11	11
query31	1761	11	11	11
query32	112	10	9	9
query33	1331	10	8	8
query34	1711	821	521	521
query35	1022	10	9	9
query36	983	10	9	9
query37	127	9	8	8
query38	3563	10	9	9
query39	1498	751	733	733
query40	319	12	10	10
query41	81	11	11	11
query42	161	10	8	8
query43	502	9	9	9
query44	1349	9	9	9
query45	377	10	9	9
query46	1202	9	9	9
query47	1777	11	8	8
query48	407	10	9	9
query49	1201	10	10	10
query50	790	9	9	9
query51	4106	10	9	9
query52	112	12	10	10
query53	229	10	10	10
query54	721	10	9	9
query55	99	10	10	10
query56	352	10	9	9
query57	1224	9	8	8
query58	378	10	9	9
query59	2651	10	9	9
query60	375	10	9	9
query61	180	10	10	10
query62	847	9	8	8
query63	275	11	11	11
query64	5664	10	9	9
query65	4048	9	8	8
query66	1649	12	11	11
query67	16846	23	14	14
query68	3532	11	10	10
query69	703	10	10	10
query70	1449	11	9	9
query71	428	412	346	346
query72	6524	10	10	10
query73	527	10	9	9
query74	9602	12	11	11
query75	3450	9	10	9
query76	2545	9	8	8
query77	1286	10	10	10
query78	9810	13	9	9
query79	1001	11	9	9
query80	656	12	10	10
query81	625	11	10	10
query82	219	9	9	9
query83	312	9	9	9
query84	285	10	8	8
query85	2025	9	9	9
query86	410	9	9	9
query87	3917	9	8	8
query88	2855	12	10	10
query89	372	11	10	10
query90	2036	9	9	9
query91	166	11	10	10
query92	80	10	10	10
query93	1069	9	9	9
query94	758	9	8	8
query95	544	11	9	9
query96	436	12	9	9
query97	3082	10	14	10
query98	227	225	239	225
query99	1516	11	9	9
Total cold run time: 288942 ms
Total hot run time: 2756 ms

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 30.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff8373a711477a675ef38eea1408dd811e5ea628, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.06
query3	0.27	0.08	0.08
query4	1.61	0.12	0.12
query5	0.27	0.26	0.25
query6	1.17	0.66	0.64
query7	0.03	0.04	0.03
query8	0.06	0.04	0.04
query9	0.61	0.52	0.50
query10	0.56	0.59	0.57
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.62	0.62	0.63
query14	1.03	1.03	1.04
query15	0.87	0.87	0.85
query16	0.40	0.39	0.41
query17	1.05	1.08	1.05
query18	0.22	0.20	0.20
query19	1.89	1.82	1.85
query20	0.01	0.02	0.02
query21	15.44	0.92	0.57
query22	0.77	1.22	0.64
query23	14.96	1.44	0.64
query24	6.90	1.29	0.76
query25	0.48	0.14	0.15
query26	0.70	0.15	0.13
query27	0.07	0.05	0.06
query28	9.81	1.35	0.95
query29	12.57	3.93	3.31
query30	0.28	0.14	0.12
query31	2.84	0.59	0.40
query32	3.25	0.57	0.48
query33	3.09	3.15	3.20
query34	16.25	5.49	4.85
query35	4.91	4.90	4.94
query36	0.68	0.54	0.51
query37	0.11	0.07	0.07
query38	0.07	0.05	0.05
query39	0.03	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.71 s
Total hot run time: 30.54 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 95.89% (140/146) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.43% (17624/33615)
Line Coverage 37.64% (160123/425380)
Region Coverage 32.18% (122141/379500)
Branch Coverage 33.53% (53554/159718)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 98.65% (146/148) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.12% (23341/32818)
Line Coverage 57.53% (244064/424224)
Region Coverage 52.93% (203545/384530)
Branch Coverage 54.54% (87471/160367)

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 26, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit d258d9b into apache:master Sep 28, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 28, 2025
Optimize Variant sparse-column scans by introducing a per-segment sparse
column cache so multiple readers/iterators reuse sparse column (keyed by
column unique_id and path).
yiguolei pushed a commit that referenced this pull request Sep 28, 2025
…56159 (#56543)

Cherry-picked from #56159

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
csun5285 added a commit to csun5285/doris that referenced this pull request Oct 9, 2025
Optimize Variant sparse-column scans by introducing a per-segment sparse
column cache so multiple readers/iterators reuse sparse column (keyed by
column unique_id and path).
csun5285 added a commit to csun5285/doris that referenced this pull request Oct 10, 2025
Optimize Variant sparse-column scans by introducing a per-segment sparse
column cache so multiple readers/iterators reuse sparse column (keyed by
column unique_id and path).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants