Skip to content

[Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsProvider #56213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zy-kkk merged 1 commit into from
Sep 19, 2025
Merged

[Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsProvider #56213

zy-kkk merged 1 commit into from
Sep 19, 2025

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Sep 19, 2025 •
edited
Loading

Changes:

Previously, only IAM Role was used (fs.s3a.assumed.role.credentials.provider).

Now supports multiple credential providers:

  • SimpleAWSCredentialsProvider

  • EnvironmentVariableCredentialsProvider

  • InstanceProfileCredentialsProvider

CREATE CATALOG `paimon_filesystem_s3_iamrole` PROPERTIES (
    "warehouse" = "s3://doris-test3/regression/paimon/",
    "iceberg.catalog.type" = "hadoop"
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1",
    "iceberg.catalog.type" = "filesystem"
   )


CREATE CATALOG `iceberg_filesystem_s3_iamrole` PROPERTIES (
    
    "type" = "iceberg",
   "iceberg.catalog.type" = "hadoop",
    "warehouse" = "s3://selectdb-qa-test3/regression/iceberg/",
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1"
    );

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Sep 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-H: Total hot run time: 1517 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 889005f96277321316d5b16bc8bc8a07ae6b17fe, data reload: false

------ Round 1 ----------------------------------
q1	17659	34	19	19
q2	2596	27	26	26
q3	10718	16	16	16
q4	11175	16	16	16
q5	8148	22	14	14
q6	224	14	13	13
q7	971	32	22	22
q8	9546	13	12	12
q9	9945	15	12	12
q10	6991	12	11	11
q11	722	25	20	20
q12	574	13	12	12
q13	18657	12	11	11
q14	242	13	12	12
q15	604	12	11	11
q16	1075	991	970	970
q17	561	12	11	11
q18	7604	12	11	11
q19	1153	13	13	13
q20	1967	367	253	253
q21	9065	21	21	21
q22	1135	13	11	11
Total cold run time: 121332 ms
Total hot run time: 1517 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	10	10	10
q2	20	20	20	20
q3	12	11	12	11
q4	13	12	11	11
q5	12	13	12	12
q6	12	12	10	10
q7	18	18	18	18
q8	10	10	10	10
q9	10	10	10	10
q10	10	10	10	10
q11	18	22	19	19
q12	11	10	9	9
q13	10	12	10	10
q14	10	11	12	11
q15	10	11	10	10
q16	1047	1074	1016	1016
q17	11	10	9	9
q18	10	9	11	9
q19	11	10	11	10
q20	1853	1992	1856	1856
q21	20	20	20	20
q22	11	10	10	10
Total cold run time: 3150 ms
Total hot run time: 3111 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-DS: Total hot run time: 2779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 889005f96277321316d5b16bc8bc8a07ae6b17fe, data reload: false

query1	1056	19	12	12
query2	7155	18	15	15
query3	7643	13	11	11
query4	26805	12	10	10
query5	4463	13	11	11
query6	394	11	11	11
query7	5417	11	10	10
query8	371	19	17	17
query9	9229	11	10	10
query10	719	12	10	10
query11	16276	12	14	12
query12	170	12	10	10
query13	1731	11	12	11
query14	10879	15	13	13
query15	218	10	9	9
query16	7158	9	9	9
query17	1452	11	9	9
query18	2892	11	9	9
query19	210	10	9	9
query20	139	11	10	10
query21	214	9	9	9
query22	4245	11	10	10
query23	33653	15	12	12
query24	9581	12	12	12
query25	726	10	9	9
query26	1049	11	10	10
query27	3405	10	9	9
query28	6349	11	10	10
query29	1227	11	10	10
query30	633	12	10	10
query31	1738	10	9	9
query32	118	11	9	9
query33	1312	11	10	10
query34	1712	820	528	528
query35	901	12	11	11
query36	1000	10	10	10
query37	118	10	9	9
query38	3571	9	9	9
query39	1544	730	739	730
query40	235	11	11	11
query41	88	12	11	11
query42	151	11	9	9
query43	498	10	9	9
query44	1369	10	10	10
query45	200	11	11	11
query46	897	10	10	10
query47	1808	10	10	10
query48	416	11	10	10
query49	1153	11	10	10
query50	756	13	11	11
query51	3961	11	10	10
query52	122	10	10	10
query53	248	11	10	10
query54	716	10	9	9
query55	91	10	9	9
query56	353	12	10	10
query57	1213	10	10	10
query58	327	12	10	10
query59	2787	10	10	10
query60	383	10	10	10
query61	204	10	9	9
query62	850	10	9	9
query63	263	10	10	10
query64	4566	12	11	11
query65	4090	10	9	9
query66	1711	11	10	10
query67	16923	33	10	10
query68	3701	10	9	9
query69	582	11	12	11
query70	1405	11	11	11
query71	441	352	346	346
query72	6944	10	9	9
query73	533	9	10	9
query74	9519	11	8	8
query75	3522	10	10	10
query76	2698	10	9	9
query77	817	11	10	10
query78	9896	13	10	10
query79	1080	10	9	9
query80	702	11	11	11
query81	685	10	10	10
query82	292	11	9	9
query83	304	12	9	9
query84	388	9	9	9
query85	1476	10	8	8
query86	602	8	9	8
query87	3955	9	8	8
query88	2858	11	10	10
query89	426	10	10	10
query90	2050	9	10	9
query91	186	9	10	9
query92	83	9	8	8
query93	2092	10	8	8
query94	2049	9	8	8
query95	521	10	9	9
query96	428	13	11	11
query97	3124	10	8	8
query98	249	235	238	235
query99	1553	9	10	9
Total cold run time: 289013 ms
Total hot run time: 2779 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

ClickBench: Total hot run time: 0.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 889005f96277321316d5b16bc8bc8a07ae6b17fe, data reload: false

query1	0.06	0.01	0.01
query2	0.11	0.01	0.00
query3	0.27	0.01	0.01
query4	1.76	0.01	0.01
query5	0.29	0.00	0.00
query6	1.64	0.01	0.00
query7	0.05	0.01	0.00
query8	0.06	0.00	0.00
query9	0.62	0.00	0.00
query10	0.60	0.00	0.00
query11	0.17	0.00	0.00
query12	0.16	0.01	0.01
query13	0.65	0.00	0.00
query14	1.03	0.00	0.01
query15	0.90	0.00	0.00
query16	0.40	0.00	0.00
query17	1.05	0.00	0.00
query18	0.22	0.00	0.00
query19	1.91	0.01	0.01
query20	0.02	0.00	0.00
query21	15.39	0.00	0.00
query22	6.12	0.00	0.00
query23	16.48	0.00	0.00
query24	1.42	0.00	0.00
query25	0.22	0.00	0.00
query26	0.16	0.00	0.00
query27	0.10	0.00	0.00
query28	1.15	0.00	0.00
query29	13.28	0.01	0.00
query30	0.32	0.01	0.00
query31	2.28	0.00	0.00
query32	5.90	0.00	0.00
query33	4.35	0.01	0.01
query34	7.45	0.00	0.00
query35	6.00	0.01	0.01
query36	0.70	0.00	0.00
query37	0.11	0.01	0.00
query38	0.08	0.00	0.00
query39	0.04	0.01	0.01
query40	0.19	0.00	0.00
query41	0.09	0.00	0.00
query42	0.07	0.00	0.00
query43	0.05	0.01	0.00
Total cold run time: 93.92 s
Total hot run time: 0.08 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

@zy-kkk zy-kkk merged commit e602e48 into apache:master Sep 19, 2025
31 of 33 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 19, 2025
@CalvinKirs
[Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsPr…
cf90997 
…ovider (#56213)

**Changes:**

Previously, only IAM Role was used
(fs.s3a.assumed.role.credentials.provider).


Now supports multiple credential providers:


- SimpleAWSCredentialsProvider

- EnvironmentVariableCredentialsProvider

- InstanceProfileCredentialsProvider
```
CREATE CATALOG `paimon_filesystem_s3_iamrole` PROPERTIES (
    "warehouse" = "s3://doris-test3/regression/paimon/",
    "iceberg.catalog.type" = "hadoop"
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1",
    "iceberg.catalog.type" = "filesystem"
   )


CREATE CATALOG `iceberg_filesystem_s3_iamrole` PROPERTIES (
    
    "type" = "iceberg",
   "iceberg.catalog.type" = "hadoop",
    "warehouse" = "s3://selectdb-qa-test3/regression/iceberg/",
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1"
    );
```
@github-actions github-actions bot mentioned this pull request Sep 19, 2025
Merged
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Sep 22, 2025
@CalvinKirs @dwdwqfwe
[Fix](Catalog)S3A: Support assuming IAM role via ProfileCredentialsPr…
60592ca 
…ovider (apache#56213)

**Changes:**

Previously, only IAM Role was used
(fs.s3a.assumed.role.credentials.provider).


Now supports multiple credential providers:


- SimpleAWSCredentialsProvider

- EnvironmentVariableCredentialsProvider

- InstanceProfileCredentialsProvider
```
CREATE CATALOG `paimon_filesystem_s3_iamrole` PROPERTIES (
    "warehouse" = "s3://doris-test3/regression/paimon/",
    "iceberg.catalog.type" = "hadoop"
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1",
    "iceberg.catalog.type" = "filesystem"
   )


CREATE CATALOG `iceberg_filesystem_s3_iamrole` PROPERTIES (
    
    "type" = "iceberg",
   "iceberg.catalog.type" = "hadoop",
    "warehouse" = "s3://selectdb-qa-test3/regression/iceberg/",
    "s3.role_arn" = "arn:aws:iam::1234554321:role/doris-bucket-role3",
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",
    "s3.region"="us-east-1"
    );
```
morrySnow pushed a commit that referenced this pull request Sep 25, 2025
@github-actions @CalvinKirs
branch-3.1: [Fix](Catalog)S3A: Support assuming IAM role via ProfileC…
98b23ae 
…redentialsProvider #56213 (#56250)

Cherry-picked from #56213

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@CalvinKirs @Thearas @doris-robot @hello-stephen @morningman @zy-kkk @morrySnow