Skip to content

branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311 #57036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 16, 2025
Merged

branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311 #57036

morningman merged 1 commit into from
Oct 16, 2025

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Oct 16, 2025 •
edited by morrySnow
Loading

picked from #56311

@CalvinKirs
[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. (apach…
2f272ad 
…e#56311)

### What problem does this PR solve?
```

CREATE CATALOG `hive_glue_iam_role` PROPERTIES (
"type" = "hms",
"hive.metastore.type" = "glue",
"glue.role_arn" = "arn:aws:iam::8888888888:role/christen",
"glue.endpoint" = "https://glue.us-east-1.amazonaws.com"
);

CREATE CATALOG `iceberg_glue_iam_role` PROPERTIES (
"type" = "iceberg",
"iceberg.catalog.type" = "glue",
"warehouse" = "s3://bucket/regression/glue/",
"glue.role_arn" = "arn:aws:iam::8888888888:role/christen",
"glue.endpoint" = "https://glue.us-east-1.amazonaws.com"
);
```

(cherry picked from commit 7202586)
@CalvinKirs CalvinKirs requested a review from morrySnow as a code owner October 16, 2025 06:11
@Thearas
Copy link
Contributor

Thearas commented Oct 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32754 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f272ad3841b5b9f7de5d569f5d5687bd499459e, data reload: false

------ Round 1 ----------------------------------
q1	17601	5644	5428	5428
q2	2028	383	281	281
q3	12226	1241	751	751
q4	10535	873	457	457
q5	9548	2418	2119	2119
q6	188	161	131	131
q7	920	754	599	599
q8	9329	1472	1175	1175
q9	5236	4985	4933	4933
q10	6769	2279	1812	1812
q11	480	285	273	273
q12	340	354	214	214
q13	17778	3640	2990	2990
q14	230	228	212	212
q15	564	475	465	465
q16	424	438	373	373
q17	598	876	366	366
q18	6972	6475	6499	6475
q19	1217	960	527	527
q20	345	352	212	212
q21	2965	2193	1979	1979
q22	1067	1041	982	982
Total cold run time: 107360 ms
Total hot run time: 32754 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5549	5521	5488	5488
q2	249	330	232	232
q3	2245	2638	2325	2325
q4	1375	1819	1452	1452
q5	4410	5064	4948	4948
q6	169	162	133	133
q7	2068	1944	1796	1796
q8	2698	2845	2678	2678
q9	7380	7284	7180	7180
q10	3067	3288	2722	2722
q11	583	504	495	495
q12	692	781	603	603
q13	3366	3853	3156	3156
q14	281	309	261	261
q15	544	484	472	472
q16	462	486	450	450
q17	1268	1778	1278	1278
q18	7685	7463	7501	7463
q19	806	917	1193	917
q20	2058	2060	1949	1949
q21	5439	5000	4484	4484
q22	1108	1068	1037	1037
Total cold run time: 53502 ms
Total hot run time: 51519 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 193443 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f272ad3841b5b9f7de5d569f5d5687bd499459e, data reload: false

query1	949	398	389	389
query2	6247	1985	1883	1883
query3	8692	208	208	208
query4	33621	24222	23496	23496
query5	3963	606	461	461
query6	303	201	186	186
query7	4209	507	327	327
query8	315	261	257	257
query9	9562	2625	2620	2620
query10	483	315	258	258
query11	18037	15368	15318	15318
query12	168	111	108	108
query13	1549	542	405	405
query14	9339	6797	7899	6797
query15	255	191	191	191
query16	8150	676	524	524
query17	1612	804	594	594
query18	2222	425	343	343
query19	231	203	171	171
query20	131	127	126	126
query21	210	131	122	122
query22	4535	4724	4360	4360
query23	35305	34489	34206	34206
query24	7628	2756	2761	2756
query25	553	496	432	432
query26	1228	298	180	180
query27	2076	509	368	368
query28	5368	2283	2248	2248
query29	790	612	489	489
query30	243	191	170	170
query31	1030	925	832	832
query32	88	64	70	64
query33	521	376	342	342
query34	780	880	533	533
query35	787	816	770	770
query36	1020	1097	978	978
query37	113	105	75	75
query38	4012	4127	4047	4047
query39	1609	1516	1486	1486
query40	212	125	110	110
query41	51	58	49	49
query42	143	112	120	112
query43	534	541	500	500
query44	1393	863	891	863
query45	201	191	181	181
query46	976	1094	720	720
query47	2030	2033	1953	1953
query48	420	462	370	370
query49	772	509	422	422
query50	727	735	476	476
query51	7528	7277	7335	7277
query52	109	109	99	99
query53	244	274	215	215
query54	568	565	492	492
query55	83	85	82	82
query56	296	284	262	262
query57	1296	1262	1227	1227
query58	258	234	234	234
query59	3195	3308	3169	3169
query60	300	286	271	271
query61	122	123	118	118
query62	828	775	720	720
query63	252	209	208	208
query64	4547	1026	662	662
query65	3393	3328	3339	3328
query66	1137	415	324	324
query67	16369	15747	15444	15444
query68	7581	861	553	553
query69	503	318	281	281
query70	1185	1155	1134	1134
query71	386	313	264	264
query72	5792	3769	3896	3769
query73	651	764	351	351
query74	10631	9344	9175	9175
query75	3223	3167	2706	2706
query76	3170	1205	784	784
query77	537	392	287	287
query78	10478	10470	9591	9591
query79	3089	916	616	616
query80	693	536	446	446
query81	505	259	220	220
query82	575	121	89	89
query83	173	167	145	145
query84	278	101	80	80
query85	805	394	308	308
query86	398	322	306	306
query87	4299	4292	4216	4216
query88	4953	2411	2402	2402
query89	412	335	296	296
query90	1824	200	192	192
query91	140	142	116	116
query92	69	60	54	54
query93	1943	899	557	557
query94	683	426	276	276
query95	349	286	277	277
query96	496	627	292	292
query97	3175	3245	3145	3145
query98	229	203	198	198
query99	1514	1473	1309	1309
Total cold run time: 296657 ms
Total hot run time: 193443 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 29.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f272ad3841b5b9f7de5d569f5d5687bd499459e, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.53	0.50	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.61	0.50	0.50
query10	0.56	0.55	0.56
query11	0.14	0.11	0.10
query12	0.13	0.11	0.12
query13	0.65	0.61	0.59
query14	0.78	0.81	0.80
query15	0.85	0.84	0.82
query16	0.39	0.38	0.37
query17	1.01	1.05	0.98
query18	0.26	0.24	0.23
query19	1.98	1.85	1.82
query20	0.02	0.01	0.02
query21	15.43	0.93	0.60
query22	0.75	0.82	0.87
query23	14.88	1.44	0.51
query24	3.65	1.58	1.16
query25	0.16	0.04	0.08
query26	0.25	0.14	0.13
query27	0.05	0.06	0.04
query28	13.88	0.97	0.42
query29	12.59	3.90	3.26
query30	0.26	0.10	0.07
query31	2.81	0.59	0.38
query32	3.22	0.53	0.46
query33	2.98	3.05	3.07
query34	16.77	5.17	4.50
query35	4.62	4.59	4.55
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 104.5 s
Total hot run time: 29.13 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 16, 2025

FE UT Coverage Report

Increment line coverage 66.18% (45/68) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. (#56311) branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311 Oct 16, 2025
@morningman morningman merged commit 92e4ff9 into apache:branch-3.1 Oct 16, 2025
25 checks passed
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@CalvinKirs @Thearas @doris-robot @hello-stephen @morrySnow @morningman