Skip to content

branch-3.1: [fix](iceberg) support reading iceberg partition table without partition value stored in file #56918 #57043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 16, 2025
Merged

branch-3.1: [fix](iceberg) support reading iceberg partition table without partition value stored in file #56918 #57043

morningman merged 1 commit into from
Oct 16, 2025
+17 −0

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Oct 16, 2025

bp: #56918

@suxiaogang223
[fix](iceberg) support reading iceberg partition table without partit…
253f64b 
…ion value stored in file (apache#56918)

Fix querying iceberg partition table when partition value is not stored
in file.
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32798 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 253f64bfcc50ff60898b89faa907ae068d08cd65, data reload: false

------ Round 1 ----------------------------------
q1	17755	5536	5538	5536
q2	2058	289	158	158
q3	10534	1292	728	728
q4	10218	874	442	442
q5	7850	2395	2157	2157
q6	183	165	133	133
q7	903	745	607	607
q8	9340	1457	1157	1157
q9	5332	4958	4946	4946
q10	6795	2287	1833	1833
q11	488	282	263	263
q12	342	356	212	212
q13	17787	3687	3006	3006
q14	225	240	240	240
q15	554	462	470	462
q16	418	430	389	389
q17	600	875	367	367
q18	7169	6462	6530	6462
q19	2469	966	541	541
q20	309	319	193	193
q21	2914	2164	1974	1974
q22	1044	1024	992	992
Total cold run time: 105287 ms
Total hot run time: 32798 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5658	5529	5496	5496
q2	235	331	228	228
q3	2255	2640	2285	2285
q4	1342	1760	1330	1330
q5	4444	4969	5169	4969
q6	179	175	132	132
q7	2107	1947	1829	1829
q8	2667	2835	2678	2678
q9	7240	7240	7202	7202
q10	2988	3280	2808	2808
q11	581	510	495	495
q12	714	763	610	610
q13	3412	3744	3222	3222
q14	288	301	279	279
q15	519	487	470	470
q16	451	496	456	456
q17	1264	1757	1257	1257
q18	7583	7410	7396	7396
q19	792	938	1162	938
q20	1994	2042	1875	1875
q21	5355	4910	4553	4553
q22	1119	1056	1026	1026
Total cold run time: 53187 ms
Total hot run time: 51534 ms

@morrySnow morrySnow changed the title branch-3.1: [fix](iceberg) support reading iceberg partition table without partition value stored in file (#56918) branch-3.1: [fix](iceberg) support reading iceberg partition table without partition value stored in file #56918 Oct 16, 2025
@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 190369 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 253f64bfcc50ff60898b89faa907ae068d08cd65, data reload: false

query1	971	388	418	388
query2	6526	1935	1927	1927
query3	6712	211	217	211
query4	34209	23669	23520	23520
query5	4330	625	428	428
query6	259	186	188	186
query7	4619	488	313	313
query8	316	267	253	253
query9	9649	2602	2597	2597
query10	484	324	270	270
query11	18389	15437	15324	15324
query12	156	112	105	105
query13	1638	535	413	413
query14	9874	6583	6666	6583
query15	207	203	185	185
query16	7905	668	516	516
query17	1803	746	560	560
query18	2218	406	335	335
query19	196	186	165	165
query20	122	123	109	109
query21	213	125	109	109
query22	4631	4579	4441	4441
query23	35446	33396	33463	33396
query24	8358	2607	2555	2555
query25	505	458	405	405
query26	1219	281	164	164
query27	2698	461	336	336
query28	5528	2160	2107	2107
query29	785	555	429	429
query30	248	182	150	150
query31	960	864	822	822
query32	86	57	56	56
query33	516	349	291	291
query34	721	831	502	502
query35	759	791	739	739
query36	986	1034	937	937
query37	103	89	68	68
query38	3963	3892	3822	3822
query39	1479	1434	1452	1434
query40	211	116	106	106
query41	53	57	50	50
query42	119	104	104	104
query43	506	523	474	474
query44	1346	809	801	801
query45	186	178	172	172
query46	864	1038	663	663
query47	1891	1940	1852	1852
query48	423	437	359	359
query49	810	515	413	413
query50	666	694	412	412
query51	7281	7241	7163	7163
query52	107	108	99	99
query53	232	251	185	185
query54	546	538	464	464
query55	80	80	79	79
query56	289	273	283	273
query57	1238	1219	1216	1216
query58	247	218	231	218
query59	2916	3055	2901	2901
query60	307	302	279	279
query61	135	132	141	132
query62	823	741	685	685
query63	232	198	191	191
query64	4669	1056	718	718
query65	3271	3184	3221	3184
query66	1083	425	335	335
query67	15936	15781	15619	15619
query68	7839	807	524	524
query69	492	308	265	265
query70	1201	1122	1100	1100
query71	373	296	259	259
query72	5502	3765	3791	3765
query73	639	748	343	343
query74	10177	9465	8986	8986
query75	3172	3124	2629	2629
query76	3166	1182	766	766
query77	492	368	273	273
query78	10509	10374	9687	9687
query79	3251	890	601	601
query80	854	530	442	442
query81	515	259	227	227
query82	694	123	93	93
query83	175	163	153	153
query84	287	107	86	86
query85	780	366	296	296
query86	400	326	297	297
query87	4417	4329	4268	4268
query88	5134	2382	2372	2372
query89	395	332	290	290
query90	1949	186	187	186
query91	137	163	109	109
query92	67	58	52	52
query93	2797	903	527	527
query94	695	388	303	303
query95	337	266	265	265
query96	496	600	286	286
query97	3195	3230	3122	3122
query98	230	213	203	203
query99	1577	1434	1296	1296
Total cold run time: 298198 ms
Total hot run time: 190369 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 28.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 253f64bfcc50ff60898b89faa907ae068d08cd65, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.52	0.52	0.52
query6	1.14	0.72	0.73
query7	0.03	0.01	0.01
query8	0.05	0.04	0.03
query9	0.57	0.50	0.50
query10	0.55	0.56	0.55
query11	0.15	0.11	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	0.76	0.81	0.80
query15	0.85	0.85	0.83
query16	0.39	0.39	0.39
query17	1.07	1.00	1.00
query18	0.24	0.23	0.22
query19	2.07	1.87	1.92
query20	0.01	0.01	0.02
query21	15.38	0.87	0.55
query22	0.74	0.86	0.58
query23	15.14	1.40	0.55
query24	3.02	1.55	0.71
query25	0.29	0.15	0.05
query26	0.18	0.16	0.14
query27	0.05	0.06	0.05
query28	13.83	1.01	0.42
query29	12.60	3.91	3.21
query30	0.27	0.10	0.06
query31	2.82	0.60	0.38
query32	3.24	0.55	0.46
query33	3.05	3.10	3.06
query34	16.82	5.22	4.58
query35	4.59	4.56	4.58
query36	0.64	0.51	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.24 s
Total hot run time: 28.63 s

@morningman morningman merged commit 51b4567 into apache:branch-3.1 Oct 16, 2025
21 of 22 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223 @hello-stephen @doris-robot @morrySnow @morningman