Skip to content

[fix](iceberg) support reading iceberg partition table without partition value stored in file #56918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Oct 15, 2025
Merged

[fix](iceberg) support reading iceberg partition table without partition value stored in file #56918

morningman merged 2 commits into from
Oct 15, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Oct 13, 2025 •
edited
Loading

What problem does this PR solve?

Fix querying iceberg partition table when partition value is not stored in file.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Oct 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 13, 2025

TPC-DS: Total hot run time: 190512 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8fdba8d3809bf197a7ce7a71f7321110287dae0, data reload: false

query1	1068	454	417	417
query2	6590	1722	1722	1722
query3	6755	224	218	218
query4	26428	23277	23470	23277
query5	5224	644	498	498
query6	329	250	227	227
query7	4656	497	295	295
query8	298	260	244	244
query9	8723	2595	2585	2585
query10	515	326	294	294
query11	15597	14984	14884	14884
query12	195	116	123	116
query13	1685	552	425	425
query14	11817	9647	9533	9533
query15	208	197	172	172
query16	7650	709	528	528
query17	1246	742	629	629
query18	2252	498	403	403
query19	289	206	204	204
query20	143	152	133	133
query21	220	137	115	115
query22	4707	4743	4542	4542
query23	34709	33534	33896	33534
query24	8453	2547	2530	2530
query25	656	591	515	515
query26	1362	279	170	170
query27	2860	515	375	375
query28	4428	2237	2178	2178
query29	873	698	566	566
query30	410	225	229	225
query31	980	859	785	785
query32	94	84	94	84
query33	1141	401	332	332
query34	840	857	535	535
query35	823	812	784	784
query36	1010	1076	968	968
query37	134	124	89	89
query38	3563	3567	3538	3538
query39	1479	1404	1386	1386
query40	213	122	113	113
query41	58	63	62	62
query42	120	108	108	108
query43	477	491	456	456
query44	1369	824	833	824
query45	186	179	172	172
query46	839	995	637	637
query47	1734	1783	1765	1765
query48	402	433	312	312
query49	764	490	406	406
query50	632	698	410	410
query51	3984	3890	3949	3890
query52	110	102	103	102
query53	232	271	197	197
query54	596	625	544	544
query55	90	93	83	83
query56	312	334	326	326
query57	1178	1188	1119	1119
query58	285	286	288	286
query59	2616	2571	2547	2547
query60	348	342	318	318
query61	165	157	164	157
query62	789	756	654	654
query63	222	187	194	187
query64	4457	1163	837	837
query65	4054	3983	3980	3980
query66	1059	439	346	346
query67	15499	15184	14994	14994
query68	8160	935	588	588
query69	510	320	294	294
query70	1475	1323	1257	1257
query71	514	336	331	331
query72	6011	4874	4883	4874
query73	657	597	365	365
query74	8995	9174	8762	8762
query75	4216	3314	2863	2863
query76	3800	1157	767	767
query77	819	420	323	323
query78	9606	9642	8864	8864
query79	2249	844	610	610
query80	710	566	516	516
query81	530	266	232	232
query82	461	160	133	133
query83	302	287	255	255
query84	292	118	92	92
query85	883	519	564	519
query86	380	325	331	325
query87	3724	3776	3734	3734
query88	3933	2248	2236	2236
query89	398	329	296	296
query90	2024	227	222	222
query91	162	171	140	140
query92	88	73	71	71
query93	1854	971	653	653
query94	690	454	345	345
query95	400	318	307	307
query96	491	567	281	281
query97	3003	2981	2863	2863
query98	242	221	209	209
query99	1469	1403	1298	1298
Total cold run time: 282248 ms
Total hot run time: 190512 ms

@doris-robot
Copy link

doris-robot commented Oct 13, 2025

ClickBench: Total hot run time: 30.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8fdba8d3809bf197a7ce7a71f7321110287dae0, data reload: false

query1	0.06	0.06	0.05
query2	0.08	0.06	0.06
query3	0.26	0.08	0.09
query4	1.62	0.12	0.12
query5	0.27	0.26	0.25
query6	1.18	0.66	0.65
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.64	0.54	0.52
query10	0.57	0.58	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.62
query14	1.03	1.03	1.02
query15	0.87	0.86	0.88
query16	0.42	0.39	0.40
query17	1.05	1.05	1.05
query18	0.22	0.20	0.21
query19	1.96	1.85	1.85
query20	0.01	0.01	0.02
query21	15.42	0.91	0.57
query22	0.77	1.31	0.71
query23	14.81	1.40	0.69
query24	7.46	1.33	1.09
query25	0.50	0.19	0.25
query26	0.64	0.15	0.16
query27	0.07	0.06	0.05
query28	9.24	1.32	0.93
query29	12.55	4.00	3.27
query30	0.28	0.14	0.11
query31	2.83	0.60	0.40
query32	3.25	0.56	0.48
query33	3.08	3.09	3.13
query34	16.07	5.46	4.81
query35	4.94	4.95	4.84
query36	0.69	0.51	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.03	0.02	0.02
query40	0.17	0.16	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.39 s
Total hot run time: 30.83 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 13, 2025

BE UT Coverage Report

Increment line coverage 23.08% (3/13) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.53% (17753/33796)
Line Coverage 37.72% (161265/427530)
Region Coverage 32.21% (123217/382494)
Branch Coverage 33.59% (54011/160792)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 23.08% (3/13) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.13% (23553/33114)
Line Coverage 57.56% (245848/427120)
Region Coverage 52.69% (204074/387326)
Branch Coverage 54.47% (88030/161607)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Oct 14, 2025

run external

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Oct 14, 2025

run buildall

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Oct 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 14, 2025

TPC-DS: Total hot run time: 190305 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 735daf029d15f4af34c6fea6ff97c9e44169e048, data reload: false

query1	1072	443	424	424
query2	6558	1693	1732	1693
query3	6754	233	220	220
query4	27015	23315	23122	23122
query5	4939	630	488	488
query6	352	233	215	215
query7	4641	497	303	303
query8	300	277	247	247
query9	8716	2553	2557	2553
query10	521	331	286	286
query11	15721	15075	15724	15075
query12	178	124	129	124
query13	1752	564	496	496
query14	11800	9686	9646	9646
query15	221	209	192	192
query16	7985	768	529	529
query17	1462	807	651	651
query18	2063	448	387	387
query19	213	202	186	186
query20	144	126	139	126
query21	221	131	117	117
query22	4816	4702	4531	4531
query23	34776	33663	33651	33651
query24	8470	2424	2416	2416
query25	569	497	441	441
query26	1240	267	155	155
query27	2734	490	355	355
query28	4377	2161	2120	2120
query29	769	587	472	472
query30	309	224	201	201
query31	904	817	729	729
query32	82	66	69	66
query33	582	372	367	367
query34	796	857	531	531
query35	793	810	742	742
query36	998	1011	926	926
query37	117	113	90	90
query38	3523	3521	3525	3521
query39	1475	1461	1413	1413
query40	214	122	114	114
query41	60	58	59	58
query42	122	111	107	107
query43	491	496	459	459
query44	1330	835	814	814
query45	185	188	178	178
query46	838	995	639	639
query47	1754	1817	1751	1751
query48	390	428	322	322
query49	756	510	464	464
query50	629	691	396	396
query51	3916	3910	3904	3904
query52	117	110	103	103
query53	236	268	195	195
query54	593	597	528	528
query55	95	83	89	83
query56	318	314	337	314
query57	1173	1192	1133	1133
query58	289	276	282	276
query59	2575	2649	2501	2501
query60	347	331	347	331
query61	161	150	165	150
query62	791	725	695	695
query63	236	207	193	193
query64	4537	1264	1008	1008
query65	4109	3957	3984	3957
query66	1073	417	347	347
query67	15515	15252	15113	15113
query68	7408	947	590	590
query69	463	319	290	290
query70	1362	1329	1306	1306
query71	495	362	317	317
query72	5707	4998	5068	4998
query73	584	607	360	360
query74	9101	9131	8744	8744
query75	3635	3432	2797	2797
query76	3411	1158	721	721
query77	773	414	334	334
query78	9572	9730	8905	8905
query79	2249	803	594	594
query80	624	558	490	490
query81	582	260	230	230
query82	218	161	135	135
query83	268	278	246	246
query84	252	116	99	99
query85	871	485	421	421
query86	385	331	298	298
query87	3754	3810	3642	3642
query88	4106	2226	2187	2187
query89	384	333	304	304
query90	2016	217	215	215
query91	157	163	134	134
query92	85	75	66	66
query93	1824	993	643	643
query94	642	445	343	343
query95	398	316	313	313
query96	491	574	284	284
query97	2917	2958	2881	2881
query98	236	217	210	210
query99	1330	1413	1268	1268
Total cold run time: 279108 ms
Total hot run time: 190305 ms

@doris-robot
Copy link

doris-robot commented Oct 14, 2025

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 735daf029d15f4af34c6fea6ff97c9e44169e048, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.29	0.28	0.25
query6	1.17	0.64	0.65
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.64	0.53	0.52
query10	0.58	0.58	0.59
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.64	0.62	0.61
query14	1.02	1.04	1.02
query15	0.89	0.83	0.88
query16	0.40	0.42	0.40
query17	1.07	1.06	1.06
query18	0.22	0.20	0.20
query19	1.95	1.89	1.89
query20	0.01	0.02	0.01
query21	15.42	0.96	0.57
query22	0.76	1.11	0.60
query23	15.15	1.40	0.67
query24	7.45	1.13	1.19
query25	0.45	0.11	0.19
query26	0.67	0.17	0.13
query27	0.07	0.06	0.05
query28	9.57	1.36	0.93
query29	12.55	3.97	3.39
query30	0.27	0.14	0.12
query31	2.84	0.59	0.39
query32	3.28	0.56	0.49
query33	3.09	3.18	3.16
query34	16.17	5.46	4.82
query35	4.91	4.95	4.87
query36	0.71	0.52	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.05
query39	0.04	0.02	0.03
query40	0.17	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 105.17 s
Total hot run time: 30.91 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 14, 2025

BE UT Coverage Report

Increment line coverage 12.50% (1/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.54% (17754/33789)
Line Coverage 37.75% (161463/427736)
Region Coverage 32.21% (123220/382585)
Branch Coverage 33.63% (54114/160933)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 12.50% (1/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.16% (23570/33124)
Line Coverage 57.58% (246154/427482)
Region Coverage 52.63% (203942/387535)
Branch Coverage 54.49% (88183/161822)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 15, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 15, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

BE Regression && UT Coverage Report

Increment line coverage 12.50% (1/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.89% (23481/33124)
Line Coverage 57.30% (244965/427482)
Region Coverage 52.34% (202855/387535)
Branch Coverage 54.22% (87733/161822)

@morningman morningman merged commit b868896 into apache:master Oct 15, 2025
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 15, 2025
@suxiaogang223
[fix](iceberg) support reading iceberg partition table without partit…
9d62d1a 
…ion value stored in file (#56918)

### What problem does this PR solve?
Fix querying iceberg partition table when partition value is not stored
in file.
@github-actions github-actions bot mentioned this pull request Oct 15, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 16, 2025
@github-actions @suxiaogang223
branch-4.0: [fix](iceberg) support reading iceberg partition table wi…
f9546e2 
…thout partition value stored in file #56918 (#56973)

Cherry-picked from #56918

Co-authored-by: Socrates <suxiaogang223@icloud.com>
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Oct 16, 2025
@suxiaogang223
[fix](iceberg) support reading iceberg partition table without partit…
253f64b 
…ion value stored in file (apache#56918)

Fix querying iceberg partition table when partition value is not stored
in file.
@suxiaogang223 suxiaogang223 mentioned this pull request Oct 16, 2025
Merged
morningman pushed a commit that referenced this pull request Oct 16, 2025
@suxiaogang223
branch-3.1: [fix](iceberg) support reading iceberg partition table wi…
51b4567 
…thout partition value stored in file #56918 (#57043)

bp: #56918
@yiguolei yiguolei mentioned this pull request Nov 5, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@suxiaogang223 @hello-stephen @doris-robot @morningman @zy-kkk @yiguolei