Skip to content

branch-3.1: [Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsProvider #56213 #56250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 25, 2025
Merged

branch-3.1: [Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsProvider #56213 #56250

morrySnow merged 1 commit into from
Sep 25, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 19, 2025

Cherry-picked from #56213

@CalvinKirs
[Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsPr…
cf90997 
…ovider (#56213)

**Changes:**

Previously, only IAM Role was used
(fs.s3a.assumed.role.credentials.provider).


Now supports multiple credential providers:


- SimpleAWSCredentialsProvider

- EnvironmentVariableCredentialsProvider

- InstanceProfileCredentialsProvider
```
CREATE CATALOG `paimon_filesystem_s3_iamrole` PROPERTIES (
    "warehouse" = "s3://doris-test3/regression/paimon/",
    "iceberg.catalog.type" = "hadoop"
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1",
    "iceberg.catalog.type" = "filesystem"
   )


CREATE CATALOG `iceberg_filesystem_s3_iamrole` PROPERTIES (
    
    "type" = "iceberg",
   "iceberg.catalog.type" = "hadoop",
    "warehouse" = "s3://selectdb-qa-test3/regression/iceberg/",
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1"
    );
```
@github-actions github-actions bot requested a review from morrySnow as a code owner September 19, 2025 10:53
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 19, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-H: Total hot run time: 32387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf90997c91fed5cc5f2c4ff132faa4320f266d80, data reload: false

------ Round 1 ----------------------------------
q1	17578	5396	5364	5364
q2	2027	398	279	279
q3	12153	1237	738	738
q4	10252	877	457	457
q5	8841	2313	2132	2132
q6	179	164	129	129
q7	914	783	618	618
q8	9331	1420	1165	1165
q9	5152	4944	4868	4868
q10	6762	2270	1800	1800
q11	474	281	262	262
q12	333	360	217	217
q13	17804	3588	2979	2979
q14	219	222	213	213
q15	529	465	467	465
q16	436	417	370	370
q17	616	857	374	374
q18	6865	6309	6254	6254
q19	1218	942	552	552
q20	324	347	207	207
q21	2785	2212	1958	1958
q22	1062	1020	986	986
Total cold run time: 105854 ms
Total hot run time: 32387 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5567	5406	5329	5329
q2	233	337	237	237
q3	2318	2643	2358	2358
q4	1418	1821	1435	1435
q5	4392	4937	4925	4925
q6	170	163	129	129
q7	2009	1951	1798	1798
q8	2568	2809	2706	2706
q9	7221	7246	7185	7185
q10	3019	3326	2764	2764
q11	562	503	491	491
q12	619	740	574	574
q13	3368	3804	3178	3178
q14	282	302	272	272
q15	508	470	482	470
q16	454	470	428	428
q17	1212	1746	1253	1253
q18	7513	7427	7348	7348
q19	784	977	1124	977
q20	2016	2079	1891	1891
q21	5277	4920	4790	4790
q22	1083	1058	1008	1008
Total cold run time: 52593 ms
Total hot run time: 51546 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-DS: Total hot run time: 192825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf90997c91fed5cc5f2c4ff132faa4320f266d80, data reload: false

query1	945	403	403	403
query2	6215	1914	1826	1826
query3	8690	196	196	196
query4	33845	23788	23904	23788
query5	3851	596	440	440
query6	289	201	182	182
query7	4218	494	320	320
query8	319	273	248	248
query9	9545	2623	2602	2602
query10	465	321	250	250
query11	17972	15662	15584	15584
query12	158	103	106	103
query13	1550	536	402	402
query14	9977	6446	7153	6446
query15	250	188	174	174
query16	7945	666	506	506
query17	1544	753	623	623
query18	2168	434	323	323
query19	226	197	169	169
query20	128	128	124	124
query21	212	131	112	112
query22	4610	4699	4508	4508
query23	35526	34371	34069	34069
query24	7409	2666	2717	2666
query25	558	486	447	447
query26	1146	293	181	181
query27	1971	493	373	373
query28	5104	2260	2233	2233
query29	803	611	462	462
query30	245	205	161	161
query31	982	890	831	831
query32	86	59	61	59
query33	501	399	323	323
query34	755	854	529	529
query35	769	826	740	740
query36	1037	1086	970	970
query37	107	97	64	64
query38	4006	4021	3998	3998
query39	1553	1463	1467	1463
query40	211	117	104	104
query41	48	48	48	48
query42	119	104	109	104
query43	517	499	475	475
query44	1314	826	839	826
query45	190	182	173	173
query46	927	1069	681	681
query47	2034	2020	2007	2007
query48	426	428	368	368
query49	743	488	401	401
query50	666	704	447	447
query51	7350	7358	7235	7235
query52	103	106	93	93
query53	226	261	192	192
query54	552	555	475	475
query55	79	82	77	77
query56	274	296	264	264
query57	1280	1285	1261	1261
query58	247	217	215	215
query59	3104	3100	2967	2967
query60	293	282	267	267
query61	118	111	114	111
query62	793	774	709	709
query63	237	199	194	194
query64	4413	1015	697	697
query65	3356	3315	3272	3272
query66	949	418	317	317
query67	16580	15781	15745	15745
query68	7619	828	555	555
query69	484	309	266	266
query70	1193	1061	1121	1061
query71	385	306	266	266
query72	5791	3804	3931	3804
query73	650	740	351	351
query74	10185	9350	8893	8893
query75	3246	3150	2668	2668
query76	3158	1178	767	767
query77	654	364	277	277
query78	10321	10383	9684	9684
query79	3856	826	610	610
query80	742	522	451	451
query81	515	269	221	221
query82	618	123	88	88
query83	181	161	147	147
query84	283	101	83	83
query85	773	357	300	300
query86	377	317	286	286
query87	4327	4338	4244	4244
query88	5012	2526	2420	2420
query89	420	339	299	299
query90	1790	189	182	182
query91	138	144	112	112
query92	61	58	53	53
query93	2181	888	566	566
query94	694	406	303	303
query95	345	288	276	276
query96	498	601	284	284
query97	3218	3300	3178	3178
query98	222	209	203	203
query99	1534	1424	1304	1304
Total cold run time: 296300 ms
Total hot run time: 192825 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

ClickBench: Total hot run time: 28.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf90997c91fed5cc5f2c4ff132faa4320f266d80, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.11
query5	0.51	0.53	0.51
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.57	0.50	0.50
query10	0.56	0.54	0.55
query11	0.14	0.11	0.11
query12	0.13	0.10	0.10
query13	0.63	0.59	0.59
query14	0.79	0.80	0.80
query15	0.83	0.84	0.82
query16	0.41	0.39	0.41
query17	1.02	0.99	1.06
query18	0.25	0.24	0.23
query19	1.84	1.86	1.85
query20	0.01	0.01	0.01
query21	15.39	0.88	0.61
query22	0.74	0.84	0.69
query23	15.05	1.42	0.56
query24	3.69	0.94	0.78
query25	0.26	0.20	0.09
query26	0.39	0.15	0.13
query27	0.05	0.05	0.05
query28	13.23	0.99	0.43
query29	12.58	3.91	3.25
query30	0.25	0.09	0.07
query31	2.83	0.63	0.38
query32	3.22	0.54	0.47
query33	2.99	3.00	3.06
query34	16.57	5.26	4.56
query35	4.58	4.59	4.57
query36	0.64	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.79 s
Total hot run time: 28.88 s

@morrySnow morrySnow merged commit 98b23ae into branch-3.1 Sep 25, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-56213-branch-3.1 branch September 25, 2025 09:23
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@hello-stephen @doris-robot @morningman @morrySnow @dataroaring @CalvinKirs