Skip to content

branch-3.1: [feat](ObjectStorage)Relax endpoint validation for private object storage #56579 #56641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 13, 2025
Merged

branch-3.1: [feat](ObjectStorage)Relax endpoint validation for private object storage #56579 #56641

morrySnow merged 1 commit into from
Oct 13, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 29, 2025

Cherry-picked from #56579

@CalvinKirs
[feat](ObjectStorage)Relax endpoint validation for private object sto…
e95be33 
…rage (#56579)

…

### 
Support private deployments by removing strict endpoint checks. If the
endpoint is non-standard, region cannot be inferred automatically, so
users must provide the region explicitly.
@github-actions github-actions bot requested a review from morrySnow as a code owner September 29, 2025 11:08
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 29, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 29, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 29, 2025

TPC-H: Total hot run time: 32925 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

------ Round 1 ----------------------------------
q1	17612	5623	5512	5512
q2	2044	420	299	299
q3	12667	1571	762	762
q4	10773	874	454	454
q5	9634	2399	2147	2147
q6	194	162	137	137
q7	904	741	617	617
q8	9346	1465	1173	1173
q9	5333	4968	4978	4968
q10	6801	2290	1834	1834
q11	489	281	263	263
q12	344	362	215	215
q13	17777	3612	3033	3033
q14	237	224	215	215
q15	529	470	458	458
q16	450	440	387	387
q17	621	883	379	379
q18	6707	6457	6339	6339
q19	1101	963	544	544
q20	327	354	219	219
q21	2905	2130	1970	1970
q22	1044	1016	1000	1000
Total cold run time: 107839 ms
Total hot run time: 32925 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5524	5509	5545	5509
q2	237	358	234	234
q3	2282	2650	2328	2328
q4	1348	1755	1342	1342
q5	4472	5122	5055	5055
q6	172	163	127	127
q7	2121	1989	1876	1876
q8	2724	2870	2797	2797
q9	7375	7310	7273	7273
q10	3051	3211	2766	2766
q11	573	528	502	502
q12	702	759	663	663
q13	3405	3829	3129	3129
q14	295	311	263	263
q15	522	468	464	464
q16	442	493	444	444
q17	1265	1754	1305	1305
q18	7630	7453	7346	7346
q19	857	1158	1089	1089
q20	2056	2091	1898	1898
q21	5359	4860	4636	4636
q22	1074	1093	1013	1013
Total cold run time: 53486 ms
Total hot run time: 52059 ms

@doris-robot
Copy link

doris-robot commented Sep 29, 2025

TPC-DS: Total hot run time: 192972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

query1	970	410	414	410
query2	6154	1929	1913	1913
query3	8690	208	198	198
query4	33745	23933	23444	23444
query5	3692	606	494	494
query6	298	188	169	169
query7	4200	502	320	320
query8	306	245	236	236
query9	9434	2622	2606	2606
query10	474	335	260	260
query11	18148	15516	15223	15223
query12	162	113	116	113
query13	1555	555	431	431
query14	9024	7242	6634	6634
query15	232	187	182	182
query16	8046	688	488	488
query17	1603	768	601	601
query18	2172	447	350	350
query19	240	198	173	173
query20	130	124	130	124
query21	206	136	115	115
query22	4522	4547	4441	4441
query23	35014	34273	34345	34273
query24	7307	2711	2712	2711
query25	543	502	435	435
query26	750	308	188	188
query27	2048	470	369	369
query28	5316	2259	2225	2225
query29	720	640	487	487
query30	244	189	164	164
query31	1057	912	866	866
query32	98	63	60	60
query33	505	379	319	319
query34	771	839	513	513
query35	789	840	747	747
query36	1003	1059	991	991
query37	104	93	74	74
query38	3975	4007	3929	3929
query39	1557	1489	1507	1489
query40	219	120	109	109
query41	52	47	47	47
query42	123	104	102	102
query43	514	512	485	485
query44	1335	833	829	829
query45	184	177	169	169
query46	885	1061	681	681
query47	1984	1990	1969	1969
query48	426	440	367	367
query49	737	495	413	413
query50	686	699	426	426
query51	7444	7225	7388	7225
query52	103	106	94	94
query53	233	259	189	189
query54	562	548	511	511
query55	81	79	80	79
query56	274	277	264	264
query57	1288	1276	1253	1253
query58	234	232	221	221
query59	3039	3160	3039	3039
query60	303	314	304	304
query61	118	122	119	119
query62	795	780	709	709
query63	237	195	210	195
query64	3373	1036	661	661
query65	3366	3328	3267	3267
query66	925	412	303	303
query67	16407	15731	15741	15731
query68	7552	839	557	557
query69	500	321	269	269
query70	1214	1102	1074	1074
query71	371	305	286	286
query72	5753	3754	3870	3754
query73	643	740	364	364
query74	10405	9185	9255	9185
query75	3212	3170	2685	2685
query76	3037	1162	775	775
query77	501	385	296	296
query78	10441	10445	9687	9687
query79	3950	891	591	591
query80	791	527	434	434
query81	530	260	224	224
query82	568	127	89	89
query83	174	161	148	148
query84	281	106	77	77
query85	767	356	314	314
query86	384	320	305	305
query87	4320	4356	4219	4219
query88	5235	2408	2395	2395
query89	410	334	295	295
query90	1864	188	184	184
query91	140	140	115	115
query92	69	53	55	53
query93	2492	912	558	558
query94	687	389	299	299
query95	348	280	265	265
query96	484	611	283	283
query97	3193	3281	3148	3148
query98	235	198	200	198
query99	1512	1426	1320	1320
Total cold run time: 293794 ms
Total hot run time: 192972 ms

@doris-robot
Copy link

doris-robot commented Sep 29, 2025

ClickBench: Total hot run time: 28.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.53	0.51	0.53
query6	1.15	0.75	0.72
query7	0.03	0.01	0.01
query8	0.05	0.03	0.03
query9	0.57	0.51	0.51
query10	0.56	0.58	0.56
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.60	0.59
query14	0.77	0.80	0.80
query15	0.85	0.83	0.83
query16	0.39	0.39	0.40
query17	1.00	1.08	1.02
query18	0.24	0.23	0.22
query19	1.98	1.87	1.91
query20	0.02	0.02	0.01
query21	15.41	0.95	0.59
query22	0.75	0.77	0.56
query23	15.25	1.45	0.58
query24	3.09	0.87	1.50
query25	0.14	0.07	0.15
query26	0.33	0.15	0.13
query27	0.05	0.05	0.05
query28	13.96	0.99	0.45
query29	12.56	4.00	3.29
query30	0.25	0.09	0.07
query31	2.82	0.58	0.37
query32	3.22	0.54	0.46
query33	3.04	3.04	3.03
query34	16.46	5.24	4.55
query35	4.50	4.58	4.59
query36	0.64	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.93 s
Total hot run time: 28.96 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 29, 2025

FE UT Coverage Report

Increment line coverage 76.92% (10/13) 🎉
Increment coverage report
Complete coverage report

@morrySnow
Copy link
Contributor

morrySnow commented Oct 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 11, 2025

TPC-H: Total hot run time: 32958 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

------ Round 1 ----------------------------------
q1	17577	5617	5418	5418
q2	2041	395	294	294
q3	12169	1285	767	767
q4	10568	877	455	455
q5	9657	2425	2191	2191
q6	195	163	129	129
q7	926	746	611	611
q8	9330	1452	1194	1194
q9	5342	4968	4939	4939
q10	6783	2280	1827	1827
q11	481	280	263	263
q12	348	358	213	213
q13	17778	3608	3027	3027
q14	225	223	217	217
q15	543	482	457	457
q16	431	434	371	371
q17	622	870	357	357
q18	6968	6468	6479	6468
q19	1201	960	558	558
q20	330	346	204	204
q21	3154	2165	2008	2008
q22	1042	1065	990	990
Total cold run time: 107711 ms
Total hot run time: 32958 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5531	5529	5528	5528
q2	244	333	231	231
q3	2248	2701	2335	2335
q4	1380	1786	1390	1390
q5	4444	5133	5066	5066
q6	169	169	127	127
q7	2227	1982	1854	1854
q8	2656	2865	2787	2787
q9	7364	7304	7365	7304
q10	3026	3266	2786	2786
q11	594	531	491	491
q12	669	774	616	616
q13	3437	3826	3191	3191
q14	300	290	281	281
q15	505	477	477	477
q16	466	492	438	438
q17	1267	1729	1246	1246
q18	7670	7485	7332	7332
q19	842	995	1126	995
q20	2066	2126	1920	1920
q21	5326	4833	4595	4595
q22	1094	1097	1081	1081
Total cold run time: 53525 ms
Total hot run time: 52071 ms

@doris-robot
Copy link

doris-robot commented Oct 11, 2025

TPC-DS: Total hot run time: 193509 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

query1	942	411	427	411
query2	6245	1902	1916	1902
query3	8692	196	193	193
query4	33390	24099	23599	23599
query5	4116	618	472	472
query6	305	196	181	181
query7	4203	493	331	331
query8	309	257	267	257
query9	9428	2601	2589	2589
query10	470	321	260	260
query11	17986	15350	15295	15295
query12	154	110	121	110
query13	1571	546	420	420
query14	9663	6694	7334	6694
query15	247	200	189	189
query16	8145	667	514	514
query17	1610	769	585	585
query18	2215	416	320	320
query19	209	190	166	166
query20	131	123	124	123
query21	207	134	108	108
query22	4742	4859	4543	4543
query23	35188	34528	34261	34261
query24	7509	2721	2742	2721
query25	544	468	414	414
query26	1177	283	172	172
query27	2066	493	362	362
query28	5127	2217	2181	2181
query29	789	615	477	477
query30	254	194	161	161
query31	1014	911	841	841
query32	103	60	61	60
query33	512	392	330	330
query34	776	863	525	525
query35	812	826	761	761
query36	1011	1043	986	986
query37	110	97	71	71
query38	4139	4026	4020	4020
query39	1533	1478	1502	1478
query40	206	127	110	110
query41	53	53	52	52
query42	139	115	108	108
query43	519	533	484	484
query44	1388	868	853	853
query45	188	179	175	175
query46	912	1064	674	674
query47	2008	2023	1976	1976
query48	444	468	366	366
query49	800	513	420	420
query50	691	693	457	457
query51	7365	7378	7294	7294
query52	104	100	94	94
query53	234	261	191	191
query54	564	558	467	467
query55	79	80	79	79
query56	280	276	254	254
query57	1301	1293	1227	1227
query58	236	223	234	223
query59	3174	3132	3051	3051
query60	290	287	289	287
query61	115	116	109	109
query62	806	745	705	705
query63	232	202	202	202
query64	4538	997	641	641
query65	3381	3330	3310	3310
query66	1081	416	309	309
query67	16543	15868	15645	15645
query68	7823	825	538	538
query69	479	302	270	270
query70	1156	1130	1109	1109
query71	352	292	262	262
query72	5758	3926	3830	3830
query73	645	751	363	363
query74	10355	9434	9411	9411
query75	3177	3192	2690	2690
query76	3019	1169	772	772
query77	469	388	295	295
query78	10540	10361	9624	9624
query79	3080	887	602	602
query80	776	520	441	441
query81	521	257	217	217
query82	735	124	90	90
query83	159	165	146	146
query84	242	100	82	82
query85	795	388	314	314
query86	430	315	295	295
query87	4267	4312	4270	4270
query88	5235	2388	2364	2364
query89	408	331	291	291
query90	1819	189	189	189
query91	161	169	109	109
query92	68	54	52	52
query93	2727	877	530	530
query94	798	417	290	290
query95	345	274	274	274
query96	496	617	281	281
query97	3160	3293	3165	3165
query98	226	203	208	203
query99	1351	1449	1294	1294
Total cold run time: 296887 ms
Total hot run time: 193509 ms

@doris-robot
Copy link

doris-robot commented Oct 11, 2025

ClickBench: Total hot run time: 29.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e95be3319170cb3d5d62c09d5a96d4ed89596202, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.11	0.10
query5	0.53	0.53	0.50
query6	1.13	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.60	0.59
query14	0.78	0.82	0.79
query15	0.85	0.83	0.82
query16	0.40	0.40	0.39
query17	1.06	1.02	1.02
query18	0.25	0.23	0.23
query19	1.97	1.90	1.91
query20	0.01	0.02	0.02
query21	15.37	0.94	0.60
query22	0.73	0.79	0.81
query23	14.97	1.36	0.52
query24	3.01	1.19	0.98
query25	0.18	0.08	0.06
query26	0.39	0.15	0.14
query27	0.04	0.04	0.04
query28	13.34	1.00	0.42
query29	12.58	3.93	3.24
query30	0.26	0.09	0.07
query31	2.81	0.59	0.38
query32	3.22	0.54	0.46
query33	2.99	3.07	3.06
query34	16.74	5.17	4.56
query35	4.56	4.53	4.56
query36	0.63	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 103.32 s
Total hot run time: 29.15 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 11, 2025

FE UT Coverage Report

Increment line coverage 76.92% (10/13) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit 26f95b2 into branch-3.1 Oct 13, 2025
23 checks passed
@morrySnow morrySnow deleted the auto-pick-56579-branch-3.1 branch October 13, 2025 04:01
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @CalvinKirs