branch-3.1: [opt](catalog) support nested namespaces of iceberg #56415 #56874 #57035
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Iceberg has 3 levels of metadata: catalog, namespace and table, mapping to Doris' catalog, database and table. Iceberg support nested namespaces, which means the following namespaces are valid: ``` ns1 ns1.ns2 ns1.ns2.ns3 ``` So we need to support mapping nested namespace to Doris' database. This PR add a global variable `enable_nested_namespace` to control this behavior. Default is `false`, and no logic is changed. If set to true, Doris can support following statments: ``` mysql> switch iceberg; mysql> show databases; +--------------------+ | Database | +--------------------+ | nested | | nested.db1 | | nested.db2 | +--------------------+ mysql> use iceberg.nested.db1; ERROR 1049 (42000): Only one dot can be in the name: iceberg.nested.db1 mysql> use iceberg.`nested.db1`; ERROR 5086 (42000): errCode = 2, detailMessage = Unknown catalog 'nested' mysql> set global enable_nested_namespace=true; mysql> use iceberg.nested.db1; Database changed mysql> select k1 from iceberg.`nested.db1`.nested1; mysql> select nested1.k1 from `nested.db1`.nested1; mysql> select `nested.db1`.nested1.k1 from iceberg.`nested.db1`.nested1; mysql> select iceberg.`nested.db1`.nested1.k1 from nested1; +------+ | k1 | +------+ | 1 | +------+ mysql> refresh catalog iceberg; mysql> refresh database iceberg.`nested.db1`; mysql> refresh table iceberg.`nested.db1`.nested1; Query OK, 0 rows affected (0.01 sec) ``` But, I can execute statement like: ``` use iceberg.`nested.db1`; ``` I don't know why, there is a very strange behavior in MySQL client, when adding back quota, the INIT_DB command can only receive `nested.db1` part, but expect `iceberg.nested.db1`. Also support creating nested database name in internal catalog: ``` create database `db1.db2` ```
Followup apache#56415 Problem Summary: 1. The previous `getNamespace` logic is wrong, we should split the `dbName` by `.` to create namespaces. 2. Allow not specify `oauth.uri` of iceberg rest catalog, to follow the new spec of IRC So we can connect Snowflake open catalog like this: ``` CREATE CATALOG ice PROPERTIES ( 'type' = 'iceberg', 'warehouse' = 'yy_external_catalog3', 'iceberg.catalog.type' = 'rest', 'iceberg.rest.uri' = 'https://xxx.snowflakecomputing.com/polaris/api/catalog', 'iceberg.rest.security.type' = 'oauth2', 'iceberg.rest.oauth2.credential' = 'id:secrete, 'iceberg.rest.oauth2.scope' = 'PRINCIPAL_ROLE:yy_sn_principal_role', 'iceberg.rest.nested-namespace-enabled' = 'true', 's3.endpoint' = 'https://s3.us-west-2.amazonaws.com', 's3.region' = 'us-west-2', 'iceberg.rest.nested-namespace-enabled' = 'true' ); ```
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32676 ms |
TPC-DS: Total hot run time: 192433 ms |
ClickBench: Total hot run time: 29.35 s |
morrySnow
approved these changes
Oct 16, 2025
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
bp #56415 #56874