Skip to content

branch-3.1: [fix](iceberg)Ensure proper authentication context before accessing Iceberg Catalog #52149 #52235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 24, 2025
Merged

branch-3.1: [fix](iceberg)Ensure proper authentication context before accessing Iceberg Catalog #52149 #52235

morrySnow merged 1 commit into from
Jun 24, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 24, 2025

Cherry-picked from #52149

@CalvinKirs @BePPPower
[fix](iceberg)Ensure proper authentication context before accessing I…
8ef4ac1 
…ceberg Catalog (#52149)

### What problem does this PR solve?
```
CREATE CATALOG `docker2_catalog3_iceberg` PROPERTIES (
"warehouse" = "hdfs://hadoop-master-2:8620/user/iceberg/hms/",
"uri" = "thrift://hadoop-master-2:9683",
"type" = "iceberg",
"list-all-tables" = "true",
"io-impl" = "org.apache.doris.datasource.iceberg.fileio.DelegateFileIO",
"iceberg.catalog.type" = "hms",
"hive.metastore.uris" = "thrift://hadoop-master-2:9683",
"hive.metastore.sasl.enabled" = "true",
"hive.metastore.kerberos.principal" = "hive/hadoop-master-2@OTHERREALM.COM",
"hadoop.security.authentication" = "kerberos",
"hadoop.kerberos.principal" = "hdfs/test4@OTHERREALM.COM",
"hadoop.kerberos.keytab" = "/kerberos/keytab/docker2/test4",
"fs.defaultFS" = "hdfs://hadoop-master-2:8620"
); 

show create create database icebergcatalog.db
```
```
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Failed to connect to Hive Metastore
        ... 13 more
Caused by: org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:85) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:70) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveCatalog.loadNamespaceMetadata(HiveCatalog.java:643) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.doris.datasource.iceberg.IcebergExternalDatabase.getLocation(IcebergExternalDatabase.java:45) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCreateDatabaseCommand.doRun(ShowCreateDatabaseCommand.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCommand.run(ShowCommand.java:54) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:769) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:112) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:189) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:70) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveCatalog.loadNamespaceMetadata(HiveCatalog.java:643) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.doris.datasource.iceberg.IcebergExternalDatabase.getLocation(IcebergExternalDatabase.java:45) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCreateDatabaseCommand.doRun(ShowCreateDatabaseCommand.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCommand.run(ShowCommand.java:54) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:769) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.GeneratedConstructorAccessor114.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:112) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:189) ~[iceberg-common-1.9.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:70) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) ~[iceberg-core-1.9.1.jar:?]
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.iceberg.hive.HiveCatalog.loadNamespaceMetadata(HiveCatalog.java:643) ~[hive-catalog-shade-3.0.1.jar:3.0.1]
        at org.apache.doris.datasource.iceberg.IcebergExternalDatabase.getLocation(IcebergExternalDatabase.java:45) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCreateDatabaseCommand.doRun(ShowCreateDatabaseCommand.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ShowCommand.run(ShowCommand.java:54) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:769) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more
```
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
@github-actions github-actions bot requested a review from morrySnow as a code owner June 24, 2025 10:16
@morrySnow morrySnow closed this Jun 24, 2025
@morrySnow morrySnow reopened this Jun 24, 2025
@morrySnow
Copy link
Contributor

morrySnow commented Jun 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-H: Total hot run time: 39795 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ef4ac1dc3b3f551dd10bf6f2e3e349e95dd9a26, data reload: false

------ Round 1 ----------------------------------
q1	17579	6942	6646	6646
q2	2044	190	175	175
q3	10529	1141	1179	1141
q4	10216	749	765	749
q5	7738	2894	2810	2810
q6	215	135	137	135
q7	972	617	618	617
q8	9371	1978	2067	1978
q9	6563	6415	6397	6397
q10	7031	2287	2306	2287
q11	458	260	259	259
q12	422	221	217	217
q13	17803	2969	2949	2949
q14	242	204	206	204
q15	511	465	474	465
q16	450	390	371	371
q17	987	563	564	563
q18	7350	6680	6545	6545
q19	1312	1082	973	973
q20	490	208	211	208
q21	3983	3190	3112	3112
q22	1079	994	1006	994
Total cold run time: 107345 ms
Total hot run time: 39795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6648	6547	6538	6538
q2	324	231	230	230
q3	2948	2754	2816	2754
q4	2050	1853	1798	1798
q5	5732	5743	5689	5689
q6	209	128	126	126
q7	2174	1786	1830	1786
q8	3349	3528	3508	3508
q9	8827	8794	8874	8794
q10	3572	3545	3541	3541
q11	591	511	498	498
q12	805	617	616	616
q13	8590	3172	3074	3074
q14	300	274	272	272
q15	521	477	466	466
q16	486	436	449	436
q17	1842	1620	1608	1608
q18	8311	7768	7781	7768
q19	1694	1669	1394	1394
q20	2139	1906	1832	1832
q21	5212	5058	4886	4886
q22	1092	1019	975	975
Total cold run time: 67416 ms
Total hot run time: 58589 ms

@morrySnow morrySnow merged commit df51a91 into branch-3.1 Jun 24, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-52149-branch-3.1 branch June 24, 2025 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morrySnow @doris-robot @CalvinKirs