Skip to content

branch-3.1: [opt](variant) add column cache for variant sparse column #56159 #56730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 11, 2025
Merged

branch-3.1: [opt](variant) add column cache for variant sparse column #56159 #56730

morrySnow merged 1 commit into from
Oct 11, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Oct 9, 2025 •
edited by morrySnow
Loading

picked from #56159

@csun5285 csun5285 requested a review from morrySnow as a code owner October 9, 2025 03:01
@Thearas
Copy link
Contributor

Thearas commented Oct 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Oct 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

TPC-H: Total hot run time: 32877 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d68d092598b99f56c5e410d033577e1847de251, data reload: false

------ Round 1 ----------------------------------
q1	17595	5460	5737	5460
q2	2035	453	281	281
q3	12026	1269	761	761
q4	10357	908	471	471
q5	9676	2436	2158	2158
q6	201	166	133	133
q7	929	750	619	619
q8	9329	1485	1180	1180
q9	5319	5038	4974	4974
q10	6774	2265	1833	1833
q11	479	287	277	277
q12	334	349	219	219
q13	17794	3613	3026	3026
q14	221	228	218	218
q15	512	463	459	459
q16	423	443	379	379
q17	597	876	365	365
q18	7004	6420	6318	6318
q19	1206	966	549	549
q20	327	354	216	216
q21	2784	2197	1991	1991
q22	1029	1064	990	990
Total cold run time: 106951 ms
Total hot run time: 32877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5571	5569	5496	5496
q2	244	332	239	239
q3	2260	2696	2342	2342
q4	1363	1788	1345	1345
q5	4415	4909	5085	4909
q6	166	163	130	130
q7	2133	1976	1878	1878
q8	2711	2874	2764	2764
q9	7294	7242	7299	7242
q10	3079	3369	2738	2738
q11	565	498	483	483
q12	664	795	623	623
q13	3418	3835	3152	3152
q14	282	321	267	267
q15	516	469	462	462
q16	450	470	454	454
q17	1236	1745	1296	1296
q18	7696	7448	7383	7383
q19	834	1160	1183	1160
q20	2033	2057	1881	1881
q21	5493	4962	4514	4514
q22	1141	1121	1025	1025
Total cold run time: 53564 ms
Total hot run time: 51783 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

TPC-DS: Total hot run time: 192286 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d68d092598b99f56c5e410d033577e1847de251, data reload: false

query1	975	403	389	389
query2	6154	1975	1888	1888
query3	8697	206	197	197
query4	34463	25400	23582	23582
query5	4423	632	474	474
query6	312	200	190	190
query7	4216	503	335	335
query8	295	240	238	238
query9	9454	2599	2594	2594
query10	474	319	266	266
query11	18118	15693	15174	15174
query12	157	107	115	107
query13	1543	547	421	421
query14	9244	7329	6682	6682
query15	251	188	185	185
query16	8100	695	498	498
query17	1527	767	560	560
query18	2151	411	313	313
query19	231	187	153	153
query20	125	122	119	119
query21	207	121	109	109
query22	4623	4559	4480	4480
query23	34961	34488	34227	34227
query24	7602	2686	2733	2686
query25	561	517	427	427
query26	1192	295	174	174
query27	2590	466	374	374
query28	5846	2239	2222	2222
query29	812	623	491	491
query30	246	193	162	162
query31	989	935	832	832
query32	92	63	67	63
query33	537	374	315	315
query34	780	863	533	533
query35	794	829	752	752
query36	1034	1052	994	994
query37	112	103	73	73
query38	4094	3987	3991	3987
query39	1566	1528	1447	1447
query40	225	126	118	118
query41	54	52	53	52
query42	121	111	104	104
query43	504	517	483	483
query44	1338	842	828	828
query45	189	184	181	181
query46	895	1078	682	682
query47	1987	1980	1950	1950
query48	423	447	354	354
query49	807	542	403	403
query50	686	708	432	432
query51	7407	7289	7316	7289
query52	101	101	93	93
query53	231	264	201	201
query54	561	551	484	484
query55	84	80	77	77
query56	268	282	253	253
query57	1290	1257	1211	1211
query58	241	235	222	222
query59	3113	3227	3038	3038
query60	310	297	265	265
query61	115	159	115	115
query62	825	731	694	694
query63	238	203	200	200
query64	4468	1003	696	696
query65	3386	3334	3283	3283
query66	1174	406	314	314
query67	16386	15848	15522	15522
query68	7617	820	553	553
query69	487	303	267	267
query70	1214	1113	1055	1055
query71	373	301	256	256
query72	5728	3615	3633	3615
query73	649	758	350	350
query74	10271	9173	9269	9173
query75	3237	3102	2646	2646
query76	3158	1182	776	776
query77	667	377	278	278
query78	10358	10413	9593	9593
query79	3709	888	591	591
query80	711	515	427	427
query81	508	263	223	223
query82	604	125	85	85
query83	171	160	145	145
query84	286	103	84	84
query85	775	369	300	300
query86	356	314	289	289
query87	4355	4328	4243	4243
query88	5104	2437	2410	2410
query89	401	334	294	294
query90	1837	191	189	189
query91	134	135	111	111
query92	64	55	54	54
query93	2141	881	543	543
query94	682	425	296	296
query95	341	276	273	273
query96	503	623	280	280
query97	3180	3295	3156	3156
query98	226	204	207	204
query99	1606	1411	1270	1270
Total cold run time: 298428 ms
Total hot run time: 192286 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

ClickBench: Total hot run time: 28.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d68d092598b99f56c5e410d033577e1847de251, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.51	0.51	0.50
query6	1.14	0.75	0.75
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.61	0.51	0.51
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.64	0.60	0.62
query14	0.80	0.80	0.82
query15	0.88	0.86	0.87
query16	0.39	0.40	0.40
query17	1.07	1.11	1.06
query18	0.24	0.22	0.22
query19	1.90	1.93	1.97
query20	0.02	0.01	0.02
query21	15.35	0.94	0.58
query22	0.75	0.90	0.72
query23	14.93	1.52	0.64
query24	3.13	1.38	0.30
query25	0.13	0.06	0.07
query26	0.35	0.16	0.13
query27	0.06	0.04	0.04
query28	12.76	1.13	0.44
query29	12.58	3.94	3.24
query30	0.26	0.09	0.07
query31	2.81	0.63	0.40
query32	3.23	0.58	0.48
query33	3.06	3.06	3.06
query34	16.72	5.33	4.62
query35	4.78	4.66	4.60
query36	0.64	0.50	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.1 s
Total hot run time: 28.88 s

@csun5285
[opt](variant) add column cache for variant sparse column (apache#56159)
f81048e 
Optimize Variant sparse-column scans by introducing a per-segment sparse
column cache so multiple readers/iterators reuse sparse column (keyed by
column unique_id and path).
@csun5285
Copy link
Contributor Author

csun5285 commented Oct 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-H: Total hot run time: 33166 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f81048e053b267e0cfe0ea296538d1d3d47c3f4b, data reload: false

------ Round 1 ----------------------------------
q1	17641	5536	5525	5525
q2	2033	400	279	279
q3	12377	1278	779	779
q4	10269	891	463	463
q5	8759	2458	2223	2223
q6	189	170	133	133
q7	921	753	631	631
q8	9366	1458	1227	1227
q9	5239	4971	4931	4931
q10	6785	2285	1859	1859
q11	467	293	263	263
q12	344	353	212	212
q13	17801	3643	3014	3014
q14	221	245	210	210
q15	545	471	478	471
q16	431	419	375	375
q17	622	888	373	373
q18	6785	6457	6351	6351
q19	1200	980	562	562
q20	339	363	229	229
q21	2837	2278	2074	2074
q22	1081	1039	982	982
Total cold run time: 106252 ms
Total hot run time: 33166 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5594	5588	5536	5536
q2	242	335	246	246
q3	2264	2659	2342	2342
q4	1437	1797	1374	1374
q5	4435	5072	5017	5017
q6	182	164	133	133
q7	2074	1978	1826	1826
q8	2685	2889	2721	2721
q9	7320	7388	7297	7297
q10	3079	3245	2774	2774
q11	580	533	533	533
q12	656	771	584	584
q13	3396	3769	3230	3230
q14	306	293	286	286
q15	530	489	466	466
q16	467	488	447	447
q17	1253	1758	1276	1276
q18	7673	7424	7482	7424
q19	883	1200	1100	1100
q20	2040	2038	1931	1931
q21	5399	4974	4503	4503
q22	1063	1084	1037	1037
Total cold run time: 53558 ms
Total hot run time: 52083 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

TPC-DS: Total hot run time: 191154 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f81048e053b267e0cfe0ea296538d1d3d47c3f4b, data reload: false

query1	951	416	404	404
query2	6113	1930	1891	1891
query3	8696	202	201	201
query4	33340	23811	23509	23509
query5	3630	611	458	458
query6	290	219	175	175
query7	4223	493	309	309
query8	303	242	236	236
query9	9336	2617	2593	2593
query10	451	327	253	253
query11	18009	15479	15228	15228
query12	165	118	110	110
query13	1552	533	430	430
query14	8957	7288	6508	6508
query15	238	194	186	186
query16	8017	632	460	460
query17	1526	789	582	582
query18	2113	427	320	320
query19	212	201	160	160
query20	122	119	118	118
query21	211	128	112	112
query22	4648	4749	4390	4390
query23	34970	34312	34540	34312
query24	7437	2730	2677	2677
query25	530	491	444	444
query26	1184	291	181	181
query27	2624	471	375	375
query28	5626	2264	2236	2236
query29	790	606	498	498
query30	259	200	160	160
query31	1213	955	832	832
query32	81	64	61	61
query33	524	376	319	319
query34	804	865	530	530
query35	837	809	730	730
query36	1016	1077	972	972
query37	113	100	75	75
query38	4046	4047	4018	4018
query39	1528	1509	1469	1469
query40	217	139	111	111
query41	56	52	48	48
query42	128	112	113	112
query43	513	528	486	486
query44	1387	869	853	853
query45	188	184	190	184
query46	879	1048	668	668
query47	2012	1980	1946	1946
query48	411	424	371	371
query49	761	488	414	414
query50	707	706	448	448
query51	7321	7363	7316	7316
query52	111	106	94	94
query53	234	258	199	199
query54	579	550	501	501
query55	81	81	85	81
query56	283	266	258	258
query57	1274	1266	1245	1245
query58	244	224	232	224
query59	3089	3148	3064	3064
query60	295	285	263	263
query61	118	115	124	115
query62	820	775	707	707
query63	235	197	195	195
query64	4501	994	652	652
query65	3413	3282	3305	3282
query66	1010	416	316	316
query67	16534	15969	15601	15601
query68	7531	820	546	546
query69	498	299	269	269
query70	1209	1146	1099	1099
query71	386	306	270	270
query72	5577	3772	2508	2508
query73	653	756	347	347
query74	10524	8973	8966	8966
query75	3287	3139	2688	2688
query76	3360	1200	777	777
query77	705	374	277	277
query78	10321	10479	9655	9655
query79	3659	826	599	599
query80	678	535	481	481
query81	510	254	219	219
query82	586	117	90	90
query83	180	163	147	147
query84	288	103	79	79
query85	808	362	296	296
query86	392	314	296	296
query87	4317	4363	4256	4256
query88	4940	2401	2372	2372
query89	412	328	289	289
query90	1853	188	190	188
query91	130	143	108	108
query92	61	56	51	51
query93	2304	906	547	547
query94	668	408	262	262
query95	365	291	268	268
query96	486	610	282	282
query97	3147	3231	3152	3152
query98	224	210	194	194
query99	1577	1391	1300	1300
Total cold run time: 295722 ms
Total hot run time: 191154 ms

@doris-robot
Copy link

doris-robot commented Oct 10, 2025

ClickBench: Total hot run time: 29.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f81048e053b267e0cfe0ea296538d1d3d47c3f4b, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.11	0.10
query5	0.53	0.52	0.52
query6	1.12	0.74	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.51
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.15	0.10	0.10
query13	0.62	0.60	0.59
query14	0.78	0.81	0.79
query15	0.84	0.84	0.81
query16	0.39	0.42	0.38
query17	1.05	1.05	1.01
query18	0.24	0.22	0.22
query19	1.96	1.80	1.88
query20	0.01	0.01	0.01
query21	15.38	0.94	0.58
query22	0.73	0.81	0.63
query23	15.13	1.40	0.62
query24	3.26	2.06	1.43
query25	0.11	0.17	0.06
query26	0.32	0.16	0.13
query27	0.05	0.04	0.04
query28	14.05	0.96	0.44
query29	12.54	3.92	3.32
query30	0.25	0.08	0.06
query31	2.83	0.59	0.40
query32	3.24	0.54	0.47
query33	3.01	3.04	3.05
query34	16.67	5.18	4.56
query35	4.57	4.63	4.57
query36	0.65	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.29 s
Total hot run time: 29.48 s

@morrySnow morrySnow changed the title branch-3.1: [opt](variant) add column cache for variant sparse column (#56159) branch-3.1: [opt](variant) add column cache for variant sparse column #56159 Oct 11, 2025
@morrySnow morrySnow merged commit afc3048 into apache:branch-3.1 Oct 11, 2025
20 of 21 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@csun5285 @Thearas @doris-robot @morrySnow