[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311

CalvinKirs · 2025-09-22T11:08:02Z

What problem does this PR solve?


CREATE CATALOG `hive_glue_iam_role` PROPERTIES (
"type" = "hms",
"hive.metastore.type" = "glue",
"glue.role_arn" = "arn:aws:iam::8888888888:role/christen",
"glue.endpoint" = "https://glue.us-east-1.amazonaws.com"
);

CREATE CATALOG `iceberg_glue_iam_role` PROPERTIES (
"type" = "iceberg",
"iceberg.catalog.type" = "glue",
"warehouse" = "s3://bucket/regression/glue/",
"glue.role_arn" = "arn:aws:iam::8888888888:role/christen",
"glue.endpoint" = "https://glue.us-east-1.amazonaws.com"
);

apache/doris-website#2912

hello-stephen · 2025-09-22T11:08:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

CalvinKirs · 2025-09-22T11:16:21Z

run buildall

doris-robot · 2025-09-22T11:39:13Z

TPC-H: Total hot run time: 1520 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 867f92b09d570cd7f2a1af5feb59b9fab5863f30, data reload: false

------ Round 1 ----------------------------------
q1	6793	28	21	21
q2	626	24	25	24
q3	905	16	16	16
q4	954	17	16	16
q5	2239	31	14	14
q6	229	15	13	13
q7	943	23	23	23
q8	1194	15	12	12
q9	17231	14	12	12
q10	4852	16	14	14
q11	451	22	21	21
q12	328	14	13	13
q13	17623	13	13	13
q14	252	13	13	13
q15	630	12	12	12
q16	1014	993	946	946
q17	549	13	12	12
q18	7844	14	12	12
q19	1508	13	12	12
q20	363	374	269	269
q21	3789	22	21	21
q22	1090	13	11	11
Total cold run time: 71407 ms
Total hot run time: 1520 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	11	13	11
q2	21	21	20	20
q3	11	10	11	10
q4	12	10	11	10
q5	11	11	12	11
q6	11	10	10	10
q7	19	19	20	19
q8	11	10	10	10
q9	11	10	10	10
q10	10	10	11	10
q11	20	19	19	19
q12	11	11	11	11
q13	11	10	10	10
q14	10	10	11	10
q15	10	11	10	10
q16	1045	1098	1028	1028
q17	12	10	10	10
q18	10	10	10	10
q19	11	10	10	10
q20	1879	1967	1831	1831
q21	20	19	18	18
q22	10	10	10	10
Total cold run time: 3178 ms
Total hot run time: 3098 ms

doris-robot · 2025-09-22T11:44:57Z

TPC-DS: Total hot run time: 2751 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 867f92b09d570cd7f2a1af5feb59b9fab5863f30, data reload: false

query1	1085	20	14	14
query2	7130	19	15	15
query3	7634	12	12	12
query4	26441	12	12	12
query5	4481	13	11	11
query6	400	12	11	11
query7	5425	11	11	11
query8	393	20	18	18
query9	9175	12	10	10
query10	734	11	11	11
query11	16028	12	11	11
query12	179	12	10	10
query13	1730	12	10	10
query14	11612	14	13	13
query15	413	12	13	12
query16	7240	10	10	10
query17	2135	10	9	9
query18	2962	11	12	11
query19	252	10	10	10
query20	155	11	11	11
query21	222	10	11	10
query22	4089	10	9	9
query23	33963	19	14	14
query24	10840	15	12	12
query25	801	10	9	9
query26	1843	11	12	11
query27	3420	11	10	10
query28	6283	12	11	11
query29	1830	11	11	11
query30	634	11	11	11
query31	1727	12	11	11
query32	119	13	10	10
query33	1336	12	10	10
query34	1645	825	543	543
query35	1042	11	10	10
query36	1001	10	10	10
query37	267	10	9	9
query38	3510	10	10	10
query39	1509	721	761	721
query40	312	11	10	10
query41	87	11	11	11
query42	146	10	9	9
query43	501	10	9	9
query44	1328	10	10	10
query45	379	12	11	11
query46	1190	10	11	10
query47	1829	11	8	8
query48	400	10	10	10
query49	1315	12	10	10
query50	789	12	10	10
query51	4029	10	9	9
query52	125	10	9	9
query53	257	12	12	12
query54	787	10	10	10
query55	94	10	10	10
query56	355	10	9	9
query57	1257	10	9	9
query58	415	11	10	10
query59	2631	10	9	9
query60	413	10	8	8
query61	184	9	8	8
query62	839	9	8	8
query63	263	11	10	10
query64	5546	10	10	10
query65	4182	12	10	10
query66	1774	12	11	11
query67	17047	38	11	11
query68	4442	10	9	9
query69	645	10	10	10
query70	1450	11	10	10
query71	603	371	335	335
query72	6811	12	10	10
query73	756	12	11	11
query74	9895	11	10	10
query75	3959	10	11	10
query76	3488	11	10	10
query77	1025	10	8	8
query78	9637	10	9	9
query79	1256	9	8	8
query80	758	9	9	9
query81	945	10	8	8
query82	366	9	8	8
query83	308	10	9	9
query84	263	9	8	8
query85	1540	10	9	9
query86	759	9	8	8
query87	3794	8	7	7
query88	2884	12	8	8
query89	389	8	7	7
query90	2153	9	8	8
query91	182	9	9	9
query92	93	11	9	9
query93	1378	9	10	9
query94	2043	9	8	8
query95	497	9	9	9
query96	423	9	8	8
query97	3003	9	8	8
query98	223	213	208	208
query99	1512	9	9	9
Total cold run time: 297639 ms
Total hot run time: 2751 ms

doris-robot · 2025-09-22T11:49:09Z

ClickBench: Total hot run time: 0.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 867f92b09d570cd7f2a1af5feb59b9fab5863f30, data reload: false

query1	0.07	0.02	0.01
query2	0.11	0.01	0.01
query3	0.27	0.01	0.01
query4	1.75	0.01	0.00
query5	0.29	0.01	0.00
query6	1.64	0.01	0.00
query7	0.04	0.00	0.00
query8	0.08	0.00	0.00
query9	0.62	0.00	0.00
query10	0.58	0.00	0.01
query11	0.18	0.01	0.01
query12	0.17	0.00	0.00
query13	0.66	0.00	0.01
query14	1.05	0.00	0.00
query15	0.92	0.00	0.00
query16	0.40	0.00	0.00
query17	1.06	0.00	0.00
query18	0.22	0.00	0.00
query19	2.26	0.00	0.00
query20	0.02	0.01	0.01
query21	15.93	0.00	0.00
query22	6.68	0.00	0.00
query23	15.85	0.01	0.00
query24	1.52	0.01	0.00
query25	0.22	0.00	0.00
query26	0.17	0.00	0.00
query27	0.12	0.01	0.00
query28	1.39	0.00	0.00
query29	13.14	0.01	0.00
query30	0.31	0.01	0.00
query31	2.24	0.00	0.00
query32	5.87	0.00	0.00
query33	4.36	0.00	0.00
query34	7.55	0.01	0.00
query35	6.53	0.00	0.00
query36	0.70	0.01	0.01
query37	0.11	0.00	0.00
query38	0.08	0.00	0.00
query39	0.05	0.01	0.01
query40	0.18	0.01	0.01
query41	0.10	0.00	0.01
query42	0.07	0.00	0.00
query43	0.05	0.00	0.00
Total cold run time: 95.61 s
Total hot run time: 0.08 s

hello-stephen · 2025-09-22T13:41:33Z

FE Regression Coverage Report

Increment line coverage 4.00% (2/50) 🎉
Increment coverage report
Complete coverage report

...rc/main/java/com/amazonaws/glue/catalog/credentials/ConfigurationAWSCredentialsProvider.java

fe/fe-core/src/main/java/org/apache/doris/datasource/property/ParamRules.java

morningman · 2025-09-22T18:29:05Z

...src/main/java/org/apache/doris/datasource/property/metastore/HMSGlueMetaStoreProperties.java

        hiveConf.set(AWS_CATALOG_CREDENTIALS_PROVIDER_FACTORY_CLASS_KEY,
                "com.amazonaws.glue.catalog.credentials.ConfigurationAWSCredentialsProviderFactory");
        hiveConf.set("hive.metastore.type", "glue");
+        setHiveConfPropertiesIfNotNull(hiveConf, AWSGlueConfig.AWS_GLUE_ACCESS_KEY, baseProperties.glueAccessKey);


No need to check if ak sk is set or not?

Since the authentication parameters are already validated during the initialization phase, we only need to perform null checks during usage.

morningman · 2025-09-22T18:30:10Z

...main/java/org/apache/doris/datasource/property/metastore/IcebergGlueMetaStoreProperties.java

+                props.put("client.credentials-provider.glue.session_token", glueProperties.glueSessionToken);
+            }
+            return;
+        }


I suggest:

if (ak, sk) { } else if (iam role) { } else { throw exception }

It’s just a personal preference—readability is much better this way, and deeply nested if/else statements aren’t very readable.

CalvinKirs · 2025-09-23T13:06:15Z

run buildall

doris-robot · 2025-09-23T13:46:13Z

ClickBench: Total hot run time: 30.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a398e818279707e29554bcae68587af2646858dc, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.09	0.09
query4	1.60	0.12	0.11
query5	0.28	0.27	0.25
query6	1.18	0.65	0.65
query7	0.03	0.03	0.03
query8	0.05	0.05	0.04
query9	0.61	0.55	0.53
query10	0.58	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.63	0.61
query14	1.04	1.05	1.02
query15	0.86	0.87	0.86
query16	0.40	0.40	0.39
query17	1.07	1.04	1.06
query18	0.22	0.20	0.20
query19	2.26	2.00	2.04
query20	0.02	0.01	0.02
query21	15.63	0.95	0.61
query22	0.77	1.21	0.71
query23	14.87	1.39	0.65
query24	6.71	1.11	0.51
query25	0.48	0.16	0.09
query26	0.62	0.15	0.14
query27	0.07	0.07	0.06
query28	9.40	1.36	0.94
query29	12.56	3.88	3.25
query30	0.29	0.13	0.10
query31	2.86	0.61	0.38
query32	3.25	0.57	0.49
query33	3.03	3.11	3.17
query34	16.14	5.44	4.84
query35	4.94	4.91	4.97
query36	0.71	0.52	0.49
query37	0.10	0.08	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.42 s
Total hot run time: 30.39 s

hello-stephen · 2025-09-23T16:08:09Z

FE Regression Coverage Report

Increment line coverage 8.93% (5/56) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-09-23T22:21:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-23T22:21:18Z

PR approved by anyone and no changes requested.

CalvinKirs · 2025-09-24T03:57:25Z

run buildall

hello-stephen · 2025-09-24T05:09:08Z

FE UT Coverage Report

Increment line coverage 66.18% (45/68) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-09-24T05:26:33Z

ClickBench: Total hot run time: 30.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0866e0b8dfd14909b4e89a7240b08cf2dc17bd3b, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.06	0.06
query3	0.25	0.08	0.09
query4	1.60	0.12	0.11
query5	0.29	0.26	0.25
query6	1.19	0.68	0.65
query7	0.04	0.03	0.03
query8	0.06	0.05	0.04
query9	0.61	0.54	0.52
query10	0.62	0.58	0.59
query11	0.17	0.12	0.12
query12	0.16	0.12	0.12
query13	0.64	0.63	0.62
query14	1.04	1.02	1.04
query15	0.86	0.87	0.87
query16	0.40	0.40	0.41
query17	1.07	1.02	1.06
query18	0.21	0.20	0.20
query19	2.26	2.07	2.03
query20	0.02	0.02	0.01
query21	15.62	0.95	0.60
query22	0.77	1.28	0.75
query23	14.74	1.40	0.69
query24	7.11	1.08	0.70
query25	0.52	0.33	0.07
query26	0.56	0.16	0.14
query27	0.06	0.06	0.06
query28	9.66	1.37	0.93
query29	12.58	3.92	3.24
query30	0.28	0.14	0.12
query31	2.83	0.59	0.40
query32	3.25	0.57	0.48
query33	3.04	3.12	3.15
query34	16.18	5.53	4.87
query35	4.92	4.90	4.88
query36	0.74	0.52	0.49
query37	0.11	0.08	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.08 s
Total hot run time: 30.69 s

hello-stephen · 2025-09-24T08:03:46Z

FE Regression Coverage Report

Increment line coverage 12.28% (7/57) 🎉
Increment coverage report
Complete coverage report

morningman

LGTM

github-actions · 2025-09-25T01:54:02Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? ``` CREATE CATALOG `hive_glue_iam_role` PROPERTIES ( "type" = "hms", "hive.metastore.type" = "glue", "glue.role_arn" = "arn:aws:iam::8888888888:role/christen", "glue.endpoint" = "https://glue.us-east-1.amazonaws.com" ); CREATE CATALOG `iceberg_glue_iam_role` PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "warehouse" = "s3://bucket/regression/glue/", "glue.role_arn" = "arn:aws:iam::8888888888:role/christen", "glue.endpoint" = "https://glue.us-east-1.amazonaws.com" ); ```

…Role. #56311 (#56418) Cherry-picked from #56311 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

…e#56311) ### What problem does this PR solve? ``` CREATE CATALOG `hive_glue_iam_role` PROPERTIES ( "type" = "hms", "hive.metastore.type" = "glue", "glue.role_arn" = "arn:aws:iam::8888888888:role/christen", "glue.endpoint" = "https://glue.us-east-1.amazonaws.com" ); CREATE CATALOG `iceberg_glue_iam_role` PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "warehouse" = "s3://bucket/regression/glue/", "glue.role_arn" = "arn:aws:iam::8888888888:role/christen", "glue.endpoint" = "https://glue.us-east-1.amazonaws.com" ); ``` (cherry picked from commit 7202586)

…Role. #56311 (#57036) picked from #56311

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole.

3b139ed

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole.

867f92b

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole.

e9583d1

morningman added the dev/3.1.x label Sep 22, 2025

morningman reviewed Sep 22, 2025

View reviewed changes

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole.

a398e81

morningman previously approved these changes Sep 23, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 23, 2025

github-actions bot added the reviewed label Sep 23, 2025

feat

0866e0b

CalvinKirs dismissed morningman’s stale review via 0866e0b September 24, 2025 03:56

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 24, 2025

morningman approved these changes Sep 25, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2025

hubgeter approved these changes Sep 25, 2025

View reviewed changes

CalvinKirs added the dev/4.0.x label Sep 25, 2025

CalvinKirs merged commit 7202586 into apache:master Sep 25, 2025
29 of 31 checks passed

github-actions bot added the dev/3.1.x-conflict label Sep 25, 2025

github-actions bot mentioned this pull request Sep 25, 2025

branch-4.0: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311 #56418

Merged

morningman pushed a commit that referenced this pull request Sep 25, 2025

branch-4.0: [feat](catalog)AWS Glue supports S3 access via IAM Assume…

f9a46e9

…Role. #56311 (#56418) Cherry-picked from #56311 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

yiguolei added dev/4.0.0-merged and removed dev/4.0.x labels Sep 27, 2025

CalvinKirs mentioned this pull request Oct 16, 2025

branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311 #57036

Merged

morningman pushed a commit that referenced this pull request Oct 16, 2025

branch-3.1: [feat](catalog)AWS Glue supports S3 access via IAM Assume…

92e4ff9

…Role. #56311 (#57036) picked from #56311

morningman added dev/3.1.2-merged and removed dev/3.1.x dev/3.1.x-conflict labels Oct 16, 2025

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311

[feat](catalog)AWS Glue supports S3 access via IAM AssumeRole. #56311

Uh oh!

Conversation

CalvinKirs commented Sep 22, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Uh oh!

hello-stephen commented Sep 22, 2025

Uh oh!

CalvinKirs commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

hello-stephen commented Sep 22, 2025

FE Regression Coverage Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

CalvinKirs Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

morningman Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

CalvinKirs Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

CalvinKirs commented Sep 23, 2025

Uh oh!

doris-robot commented Sep 23, 2025

Uh oh!

hello-stephen commented Sep 23, 2025

FE Regression Coverage Report

Uh oh!

github-actions bot commented Sep 23, 2025

Uh oh!

github-actions bot commented Sep 23, 2025

Uh oh!

CalvinKirs commented Sep 24, 2025

Uh oh!

hello-stephen commented Sep 24, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Sep 24, 2025

Uh oh!

hello-stephen commented Sep 24, 2025

FE Regression Coverage Report

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CalvinKirs commented Sep 22, 2025 •

edited by morningman

Loading