Skip to content

[fix](iceberg) Fix classloader usage in the Iceberg system table scanner #56150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Sep 19, 2025
Merged

[fix](iceberg) Fix classloader usage in the Iceberg system table scanner #56150

morningman merged 1 commit into from
Sep 19, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Sep 17, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Wrap open/getNext/close/nextScanTask with ThreadClassLoaderContext to keep the
Iceberg and AWS dependencies on the plugin classloader, preventing reflection f
ailures when execution switches threads.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 17, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-H: Total hot run time: 35014 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32c2ffd54aab78b58dfbe327e22ee6df9b80914e, data reload: false

------ Round 1 ----------------------------------
q1	17642	5217	5287	5217
q2	2041	333	216	216
q3	10228	1319	715	715
q4	10233	1015	561	561
q5	7526	2403	2448	2403
q6	181	166	135	135
q7	943	782	626	626
q8	9353	1290	1158	1158
q9	6950	5156	5152	5152
q10	6915	2393	1935	1935
q11	504	294	295	294
q12	356	368	228	228
q13	17785	3663	3003	3003
q14	242	258	215	215
q15	592	510	487	487
q16	1018	1018	955	955
q17	619	860	355	355
q18	7764	7206	7140	7140
q19	1242	959	585	585
q20	349	355	241	241
q21	3744	3200	2388	2388
q22	1053	1041	1005	1005
Total cold run time: 107280 ms
Total hot run time: 35014 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5164	5286	5111	5111
q2	262	340	247	247
q3	2205	2673	2314	2314
q4	1377	1731	1340	1340
q5	4219	4703	4576	4576
q6	223	179	133	133
q7	2064	2008	1856	1856
q8	2672	2593	2639	2593
q9	7368	7350	7409	7350
q10	3077	3385	2869	2869
q11	627	555	543	543
q12	720	788	684	684
q13	3586	3869	3417	3417
q14	311	344	295	295
q15	528	492	490	490
q16	1071	1106	1094	1094
q17	1233	1591	1410	1410
q18	8000	7756	7582	7582
q19	815	884	1104	884
q20	1987	2135	1975	1975
q21	4811	4402	4286	4286
q22	1112	1036	1007	1007
Total cold run time: 53432 ms
Total hot run time: 52056 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

TPC-DS: Total hot run time: 188869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32c2ffd54aab78b58dfbe327e22ee6df9b80914e, data reload: false

query1	1066	427	406	406
query2	6565	1719	1728	1719
query3	6747	226	224	224
query4	26824	23576	23673	23576
query5	4348	631	473	473
query6	339	253	222	222
query7	4645	532	305	305
query8	329	280	257	257
query9	8668	2643	2635	2635
query10	468	332	284	284
query11	15681	14989	14839	14839
query12	172	117	112	112
query13	1682	572	453	453
query14	10269	9212	9257	9212
query15	209	199	179	179
query16	7315	691	520	520
query17	1224	738	675	675
query18	2033	432	345	345
query19	200	191	173	173
query20	138	129	128	128
query21	216	135	116	116
query22	4175	4336	4162	4162
query23	33890	33070	32672	32672
query24	8619	2389	2440	2389
query25	577	503	442	442
query26	1244	276	159	159
query27	2743	512	366	366
query28	4393	2247	2224	2224
query29	834	624	482	482
query30	303	229	200	200
query31	875	836	719	719
query32	81	70	79	70
query33	603	386	364	364
query34	810	855	525	525
query35	855	839	747	747
query36	993	1000	924	924
query37	119	112	83	83
query38	3529	3569	3478	3478
query39	1512	1442	1403	1403
query40	224	126	119	119
query41	72	62	65	62
query42	128	114	111	111
query43	517	498	469	469
query44	1330	847	874	847
query45	184	179	167	167
query46	852	1023	652	652
query47	1790	1800	1730	1730
query48	395	422	325	325
query49	768	523	419	419
query50	658	686	403	403
query51	3873	4076	3908	3908
query52	109	110	102	102
query53	236	255	199	199
query54	597	583	529	529
query55	85	84	80	80
query56	322	322	306	306
query57	1210	1213	1122	1122
query58	298	266	264	264
query59	2643	2676	2569	2569
query60	338	342	330	330
query61	165	167	163	163
query62	803	729	663	663
query63	228	193	190	190
query64	4544	1187	814	814
query65	4027	3964	3953	3953
query66	1167	425	342	342
query67	15465	15366	15248	15248
query68	7986	946	588	588
query69	497	326	338	326
query70	1410	1278	1239	1239
query71	567	337	305	305
query72	6035	5010	5057	5010
query73	710	632	358	358
query74	9134	8856	8926	8856
query75	3961	3299	2775	2775
query76	3577	1179	763	763
query77	805	410	307	307
query78	9671	9669	8895	8895
query79	2005	839	582	582
query80	621	553	506	506
query81	488	265	229	229
query82	495	167	137	137
query83	273	268	239	239
query84	257	111	96	96
query85	993	465	428	428
query86	336	311	298	298
query87	3705	3702	3662	3662
query88	3695	2245	2247	2245
query89	396	328	291	291
query90	1965	213	213	213
query91	164	165	135	135
query92	82	65	60	60
query93	1683	991	660	660
query94	678	432	339	339
query95	395	323	312	312
query96	496	563	276	276
query97	2909	2981	2858	2858
query98	237	215	210	210
query99	1363	1413	1323	1323
Total cold run time: 275910 ms
Total hot run time: 188869 ms

@doris-robot
Copy link

doris-robot commented Sep 17, 2025

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32c2ffd54aab78b58dfbe327e22ee6df9b80914e, data reload: false

query1	0.06	0.05	0.06
query2	0.10	0.06	0.06
query3	0.26	0.09	0.10
query4	1.61	0.11	0.13
query5	0.29	0.27	0.26
query6	1.15	0.66	0.66
query7	0.03	0.03	0.03
query8	0.07	0.05	0.05
query9	0.58	0.55	0.53
query10	0.57	0.58	0.58
query11	0.16	0.11	0.11
query12	0.18	0.13	0.13
query13	0.63	0.63	0.63
query14	1.02	1.02	1.05
query15	0.86	0.86	0.85
query16	0.41	0.41	0.40
query17	1.06	1.06	1.06
query18	0.22	0.20	0.21
query19	1.87	1.85	1.82
query20	0.02	0.01	0.02
query21	15.42	0.91	0.58
query22	0.77	1.14	0.70
query23	14.96	1.36	0.64
query24	6.88	1.42	0.82
query25	0.49	0.17	0.10
query26	0.62	0.15	0.15
query27	0.08	0.06	0.06
query28	10.09	0.87	0.43
query29	12.57	4.00	3.25
query30	0.30	0.17	0.14
query31	2.84	0.61	0.39
query32	3.24	0.56	0.47
query33	3.01	3.10	3.10
query34	16.10	5.47	4.85
query35	4.91	4.89	4.89
query36	0.72	0.52	0.51
query37	0.11	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.04
query40	0.18	0.16	0.15
query41	0.09	0.04	0.04
query42	0.04	0.03	0.04
query43	0.04	0.04	0.04
Total cold run time: 104.72 s
Total hot run time: 30.14 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 17, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 84cb785 into apache:master Sep 19, 2025
28 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 19, 2025
@zy-kkk
[fix](iceberg) Fix classloader usage in the Iceberg system table scan…
4ed0594 
…ner (#56150)
@github-actions github-actions bot mentioned this pull request Sep 19, 2025
Merged
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Sep 22, 2025
@zy-kkk @dwdwqfwe
[fix](iceberg) Fix classloader usage in the Iceberg system table scan…
5b6b435 
…ner (apache#56150)
morrySnow pushed a commit that referenced this pull request Sep 25, 2025
@github-actions @zy-kkk
branch-3.1: [fix](iceberg) Fix classloader usage in the Iceberg syste…
6ff6fe3 
…m table scanner #56150 (#56220)

Cherry-picked from #56150

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zy-kkk @hello-stephen @doris-robot @morningman @CalvinKirs @morrySnow