Skip to content

Conversation

@CalvinKirs
Copy link
Member

Support private deployments by removing strict endpoint checks. If the endpoint is non-standard, region cannot be inferred automatically, so users must provide the region explicitly.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…rage

Body:

vbnet
Copy code
Support private deployments by removing strict endpoint checks.
If the endpoint is non-standard, region cannot be inferred automatically,
so users must provide the region explicitly.
@Thearas
Copy link
Contributor

Thearas commented Sep 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 30.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cc455bcaebaf11bd805b8fb22991d5c0435b555, data reload: false

query1	0.06	0.06	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.09
query4	1.60	0.12	0.12
query5	0.27	0.26	0.25
query6	1.19	0.65	0.66
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.53	0.51
query10	0.59	0.56	0.58
query11	0.17	0.12	0.11
query12	0.15	0.12	0.13
query13	0.63	0.63	0.63
query14	1.03	1.04	1.04
query15	0.85	0.87	0.87
query16	0.42	0.40	0.40
query17	1.06	1.04	1.06
query18	0.21	0.20	0.20
query19	2.02	1.81	1.83
query20	0.02	0.01	0.01
query21	15.43	0.93	0.58
query22	0.77	1.22	0.65
query23	14.92	1.37	0.67
query24	7.65	0.87	0.56
query25	0.54	0.24	0.07
query26	0.62	0.17	0.14
query27	0.07	0.06	0.06
query28	9.36	1.38	0.96
query29	12.56	4.01	3.28
query30	0.28	0.17	0.11
query31	2.84	0.58	0.38
query32	3.25	0.57	0.49
query33	3.16	3.13	3.21
query34	16.20	5.46	4.87
query35	4.95	4.96	4.93
query36	0.71	0.51	0.51
query37	0.10	0.07	0.08
query38	0.06	0.04	0.04
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.18 s
Total hot run time: 30.35 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 30.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3a039451fb5bc246d27f01582171434093c5792, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.06
query3	0.26	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.27	0.26
query6	1.15	0.66	0.65
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.62	0.57	0.53
query10	0.58	0.57	0.58
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.64	0.63	0.63
query14	1.03	1.04	1.03
query15	0.86	0.86	0.88
query16	0.40	0.41	0.41
query17	1.08	1.06	1.08
query18	0.22	0.21	0.19
query19	1.92	1.87	1.84
query20	0.02	0.02	0.01
query21	15.43	0.89	0.57
query22	0.75	1.16	0.68
query23	14.98	1.38	0.67
query24	7.21	2.04	0.59
query25	0.54	0.27	0.07
query26	0.69	0.16	0.13
query27	0.07	0.04	0.04
query28	8.97	1.36	0.94
query29	12.53	3.89	3.23
query30	0.30	0.14	0.11
query31	2.85	0.59	0.38
query32	3.24	0.57	0.49
query33	3.18	3.07	3.19
query34	16.24	5.46	4.94
query35	4.95	4.95	4.91
query36	0.73	0.50	0.51
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.41 s
Total hot run time: 30.32 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Copy link
Member Author

run buildall

@morningman morningman requested a review from Copilot September 29, 2025 03:33
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR relaxes endpoint validation for object storage systems to support private deployments. The change removes strict endpoint pattern matching and instead requires users to explicitly provide the region when using non-standard endpoints.

  • Removes strict endpoint validation logic that was blocking private deployments
  • Updates error handling to require region specification instead of endpoint validation
  • Modifies test cases to reflect the new validation behavior

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test_domain_connection_and_ak_sk_correction.groovy Updates error message assertion for connection failures
S3PropertiesTest.java Removes endpoint validation tests and adds region requirement tests
OSSPropertiesTest.java Changes endpoint validation to region requirement validation
OBSPropertyTest.java Updates error message from endpoint validation to region requirement
COSPropertiesTest.java Removes endpoint validation assertion
IcebergS3TablesMetaStorePropertiesTest.java Updates test to check for region requirement earlier in validation
S3Properties.java Overrides endpoint check requirement to disable validation
OSSProperties.java Updates endpoint handling logic for non-standard endpoints
AbstractS3CompatibleProperties.java Replaces endpoint validation with region requirement checks

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c1c11133cf7c2f7e1182e697cf21529afc03744, data reload: false

query1	0.06	0.05	0.04
query2	0.09	0.05	0.06
query3	0.25	0.08	0.08
query4	1.62	0.11	0.12
query5	0.28	0.27	0.25
query6	1.16	0.67	0.65
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.53	0.55
query10	0.58	0.58	0.58
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.61
query14	1.01	1.06	1.02
query15	0.88	0.87	0.87
query16	0.40	0.40	0.40
query17	1.06	1.05	1.08
query18	0.22	0.20	0.21
query19	1.91	1.89	1.89
query20	0.02	0.01	0.01
query21	15.44	0.93	0.56
query22	0.77	1.22	0.75
query23	14.78	1.35	0.66
query24	7.25	2.30	0.56
query25	0.48	0.13	0.15
query26	0.58	0.16	0.15
query27	0.07	0.06	0.05
query28	9.25	1.37	0.92
query29	12.54	3.90	3.23
query30	0.28	0.14	0.11
query31	2.87	0.61	0.39
query32	3.24	0.56	0.48
query33	3.10	3.15	3.07
query34	16.27	5.46	4.90
query35	4.98	4.88	4.92
query36	0.69	0.52	0.53
query37	0.10	0.08	0.07
query38	0.07	0.05	0.05
query39	0.03	0.04	0.03
query40	0.18	0.15	0.15
query41	0.08	0.04	0.03
query42	0.04	0.04	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.34 s
Total hot run time: 30.39 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 53.33% (8/15) 🎉
Increment coverage report
Complete coverage report

throw new IllegalArgumentException("Both the access key and the secret key must be set.");
}
if (StringUtils.isBlank(getRegion())) {
throw new IllegalArgumentException("Region must be set.");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

endpoint must be stand

@CalvinKirs
Copy link
Member Author

run buildall

zy-kkk
zy-kkk previously approved these changes Sep 29, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 29, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 190336 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc8c02b6a3a49e04834d825717770afb979ebc44, data reload: false

query1	1059	426	400	400
query2	6581	1723	1681	1681
query3	6776	224	220	220
query4	26087	23767	23873	23767
query5	5885	648	479	479
query6	351	268	246	246
query7	4661	499	305	305
query8	328	304	268	268
query9	8721	2591	2578	2578
query10	553	347	298	298
query11	15661	15117	14944	14944
query12	183	119	121	119
query13	1692	626	469	469
query14	11798	9427	9560	9427
query15	263	189	179	179
query16	8715	742	509	509
query17	1823	849	741	741
query18	2245	461	362	362
query19	315	228	184	184
query20	141	136	136	136
query21	220	143	115	115
query22	4743	4619	4621	4619
query23	34921	33429	33736	33429
query24	8379	2474	2635	2474
query25	583	552	482	482
query26	1192	283	172	172
query27	2736	518	376	376
query28	4298	2226	2220	2220
query29	792	644	476	476
query30	299	226	200	200
query31	900	806	723	723
query32	81	73	78	73
query33	575	361	342	342
query34	787	845	512	512
query35	813	848	783	783
query36	970	1027	940	940
query37	117	109	87	87
query38	3512	3562	3491	3491
query39	1503	1461	1406	1406
query40	220	128	118	118
query41	61	61	59	59
query42	123	121	114	114
query43	509	504	475	475
query44	1351	865	828	828
query45	184	181	172	172
query46	839	989	645	645
query47	1759	1783	1704	1704
query48	393	451	303	303
query49	770	510	428	428
query50	637	686	411	411
query51	4003	4005	3834	3834
query52	117	111	107	107
query53	237	267	202	202
query54	615	601	545	545
query55	88	84	87	84
query56	319	327	306	306
query57	1167	1186	1103	1103
query58	297	278	273	273
query59	2646	2545	2576	2545
query60	343	349	351	349
query61	162	160	155	155
query62	782	733	656	656
query63	230	202	199	199
query64	4445	1162	842	842
query65	4073	3998	3971	3971
query66	1025	426	339	339
query67	15454	15327	15032	15032
query68	9391	941	598	598
query69	527	341	292	292
query70	1395	1248	1311	1248
query71	510	350	313	313
query72	5788	4939	5091	4939
query73	682	567	358	358
query74	9118	9188	8679	8679
query75	4665	3312	2842	2842
query76	3825	1188	784	784
query77	1016	421	307	307
query78	9604	9904	8852	8852
query79	2182	850	610	610
query80	654	622	523	523
query81	515	264	228	228
query82	465	159	131	131
query83	298	269	245	245
query84	308	116	96	96
query85	878	487	429	429
query86	338	313	305	305
query87	3813	3733	3592	3592
query88	3419	2270	2255	2255
query89	402	326	306	306
query90	2044	233	223	223
query91	168	176	135	135
query92	92	78	67	67
query93	1890	1006	644	644
query94	683	459	348	348
query95	405	319	321	319
query96	483	591	293	293
query97	2920	2991	2871	2871
query98	248	215	207	207
query99	1477	1425	1266	1266
Total cold run time: 284311 ms
Total hot run time: 190336 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc8c02b6a3a49e04834d825717770afb979ebc44, data reload: false

query1	0.06	0.06	0.05
query2	0.09	0.05	0.06
query3	0.26	0.08	0.08
query4	1.61	0.11	0.12
query5	0.27	0.26	0.25
query6	1.18	0.64	0.64
query7	0.04	0.03	0.03
query8	0.05	0.05	0.04
query9	0.62	0.52	0.51
query10	0.58	0.57	0.58
query11	0.20	0.12	0.11
query12	0.15	0.12	0.11
query13	0.64	0.63	0.61
query14	1.04	1.05	1.03
query15	0.87	0.87	0.86
query16	0.41	0.40	0.41
query17	1.06	1.07	1.05
query18	0.22	0.20	0.20
query19	2.00	1.84	1.88
query20	0.02	0.01	0.02
query21	15.47	0.95	0.59
query22	0.78	1.06	0.97
query23	14.72	1.41	0.64
query24	7.11	1.64	0.67
query25	0.46	0.22	0.09
query26	0.58	0.17	0.13
query27	0.09	0.06	0.06
query28	9.42	1.39	0.93
query29	12.57	4.03	3.28
query30	0.28	0.13	0.11
query31	2.82	0.62	0.39
query32	3.26	0.55	0.47
query33	3.15	3.14	3.09
query34	16.14	5.45	4.86
query35	4.93	4.84	4.89
query36	0.70	0.51	0.50
query37	0.11	0.08	0.07
query38	0.07	0.05	0.04
query39	0.04	0.02	0.02
query40	0.19	0.14	0.15
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.05	0.04	0.04
Total cold run time: 104.44 s
Total hot run time: 30.53 s

@CalvinKirs
Copy link
Member Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 29, 2025
@doris-robot
Copy link

TPC-DS: Total hot run time: 190611 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 640e9869de503ac09b35f8e767788880c5b6c7e6, data reload: false

query1	1063	438	409	409
query2	6569	1708	1678	1678
query3	6751	226	217	217
query4	26858	23830	23552	23552
query5	5615	639	479	479
query6	351	248	229	229
query7	4662	492	306	306
query8	312	271	258	258
query9	8727	2556	2551	2551
query10	543	347	280	280
query11	15588	15794	15469	15469
query12	191	115	126	115
query13	2568	570	425	425
query14	12741	9616	9501	9501
query15	273	202	185	185
query16	8022	681	524	524
query17	1919	800	634	634
query18	2732	405	346	346
query19	304	233	175	175
query20	141	133	142	133
query21	215	134	114	114
query22	4534	5027	4651	4651
query23	34653	34039	33569	33569
query24	8138	2507	2499	2499
query25	588	545	471	471
query26	1572	275	168	168
query27	2730	500	350	350
query28	4327	2190	2148	2148
query29	790	623	529	529
query30	291	236	200	200
query31	1045	808	740	740
query32	81	65	73	65
query33	577	365	315	315
query34	803	860	510	510
query35	766	823	743	743
query36	976	1003	951	951
query37	122	109	85	85
query38	3458	3501	3506	3501
query39	1512	1417	1408	1408
query40	217	121	110	110
query41	59	59	59	59
query42	117	106	106	106
query43	476	500	491	491
query44	1308	819	834	819
query45	181	180	177	177
query46	825	1000	626	626
query47	1804	1850	1749	1749
query48	396	436	310	310
query49	760	482	413	413
query50	624	697	409	409
query51	3905	3949	3917	3917
query52	107	109	99	99
query53	234	259	201	201
query54	596	583	520	520
query55	88	80	86	80
query56	329	315	315	315
query57	1197	1189	1128	1128
query58	281	273	278	273
query59	2581	2659	2539	2539
query60	359	355	330	330
query61	182	155	146	146
query62	798	729	668	668
query63	231	194	192	192
query64	4399	1122	832	832
query65	4051	3921	3993	3921
query66	1048	429	329	329
query67	15551	15347	15186	15186
query68	8623	939	601	601
query69	491	332	285	285
query70	1427	1244	1319	1244
query71	486	353	316	316
query72	5830	4943	4879	4879
query73	576	584	349	349
query74	8900	9223	8621	8621
query75	4207	3341	2875	2875
query76	3381	1143	731	731
query77	843	410	318	318
query78	9740	9879	8876	8876
query79	1434	852	603	603
query80	652	565	496	496
query81	494	264	233	233
query82	439	159	133	133
query83	299	329	255	255
query84	305	107	90	90
query85	882	468	417	417
query86	348	327	299	299
query87	3753	3769	3644	3644
query88	2938	2323	2240	2240
query89	395	330	293	293
query90	2057	221	218	218
query91	165	174	132	132
query92	90	72	67	67
query93	1197	1002	637	637
query94	685	437	327	327
query95	400	327	317	317
query96	491	581	287	287
query97	2943	2973	2884	2884
query98	231	216	211	211
query99	1466	1417	1305	1305
Total cold run time: 281980 ms
Total hot run time: 190611 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 640e9869de503ac09b35f8e767788880c5b6c7e6, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.27	0.26	0.25
query6	1.18	0.66	0.65
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.63	0.53	0.52
query10	0.59	0.58	0.58
query11	0.19	0.12	0.11
query12	0.16	0.12	0.13
query13	0.64	0.62	0.63
query14	1.02	1.04	1.04
query15	0.87	0.85	0.88
query16	0.41	0.42	0.40
query17	1.07	1.07	1.07
query18	0.22	0.20	0.20
query19	1.98	1.82	1.88
query20	0.02	0.01	0.02
query21	15.42	0.94	0.57
query22	0.75	1.21	0.74
query23	14.85	1.41	0.64
query24	7.45	1.61	0.78
query25	0.50	0.22	0.07
query26	0.60	0.17	0.13
query27	0.08	0.05	0.06
query28	9.61	1.38	0.95
query29	12.64	3.99	3.32
query30	0.28	0.14	0.12
query31	2.83	0.62	0.39
query32	3.25	0.56	0.47
query33	3.07	3.14	3.30
query34	16.25	5.44	4.89
query35	4.92	4.94	4.95
query36	0.70	0.53	0.51
query37	0.10	0.07	0.07
query38	0.07	0.06	0.05
query39	0.03	0.02	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.04
query43	0.05	0.03	0.03
Total cold run time: 105.11 s
Total hot run time: 30.7 s

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 29, 2025
@CalvinKirs CalvinKirs merged commit 086eb62 into apache:master Sep 29, 2025
26 of 27 checks passed
@CalvinKirs CalvinKirs deleted the master-endpoint-check branch September 29, 2025 11:06
github-actions bot pushed a commit that referenced this pull request Sep 29, 2025
…rage (#56579)

…

### 
Support private deployments by removing strict endpoint checks. If the
endpoint is non-standard, region cannot be inferred automatically, so
users must provide the region explicitly.
github-actions bot pushed a commit that referenced this pull request Sep 29, 2025
…rage (#56579)

…

### 
Support private deployments by removing strict endpoint checks. If the
endpoint is non-standard, region cannot be inferred automatically, so
users must provide the region explicitly.
yiguolei pushed a commit that referenced this pull request Sep 30, 2025
…e object storage #56579 (#56642)

Cherry-picked from #56579

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
morrySnow pushed a commit that referenced this pull request Oct 13, 2025
…e object storage #56579 (#56641)

Cherry-picked from #56579

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
liaoxin01 pushed a commit that referenced this pull request Dec 2, 2025
#58572)

Related PR: ##56579

Problem Summary:

cuz we add the check : `Pattern STANDARD_ENDPOINT_PATTERN = Pattern
.compile("^(?:https?://)?(?:s3\\.)?oss-([a-z0-9-]+?)(?:-internal)?\\.aliyuncs\\.com$");`
the case will be invalid
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
#58572)

Related PR: ##56579

Problem Summary:

cuz we add the check : `Pattern STANDARD_ENDPOINT_PATTERN = Pattern
.compile("^(?:https?://)?(?:s3\\.)?oss-([a-z0-9-]+?)(?:-internal)?\\.aliyuncs\\.com$");`
the case will be invalid
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
#58572)

Related PR: ##56579

Problem Summary:

cuz we add the check : `Pattern STANDARD_ENDPOINT_PATTERN = Pattern
.compile("^(?:https?://)?(?:s3\\.)?oss-([a-z0-9-]+?)(?:-internal)?\\.aliyuncs\\.com$");`
the case will be invalid
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
apache#58572)

Related PR: #apache#56579

Problem Summary:

cuz we add the check : `Pattern STANDARD_ENDPOINT_PATTERN = Pattern
.compile("^(?:https?://)?(?:s3\\.)?oss-([a-z0-9-]+?)(?:-internal)?\\.aliyuncs\\.com$");`
the case will be invalid
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged dev/4.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants