Skip to content

Antalya 26.1: Expose IcebergS3 partition_key and sorting_key in system.tables#1432

Merged
zvonand merged 4 commits intoantalya-26.1from
forwardports/antalya-26.1/1095
Feb 27, 2026
Merged

Antalya 26.1: Expose IcebergS3 partition_key and sorting_key in system.tables#1432
zvonand merged 4 commits intoantalya-26.1from
forwardports/antalya-26.1/1095

Conversation

@zvonand
Copy link
Copy Markdown
Collaborator

@zvonand zvonand commented Feb 19, 2026

Supersedes #1095, #1098, #1211

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • Tiered Storage (2h)

Enmk and others added 4 commits February 19, 2026 13:50
…al_columns_in_system_tables

Antalya 25.8: Port of #959 and #1026 iceberg addtional columns in system tables
@zvonand zvonand added port-antalya PRs to be ported to all new Antalya releases antalya-26.1 labels Feb 19, 2026
@github-actions
Copy link
Copy Markdown

Workflow [PR], commit [fa81315]

@svb-alt svb-alt added the port-forward Needs to be ported to every future minor release of this major version label Feb 24, 2026
@zvonand zvonand merged commit 7fa4087 into antalya-26.1 Feb 27, 2026
671 of 678 checks passed
@alsugiliazova
Copy link
Copy Markdown
Member

alsugiliazova commented Mar 2, 2026

PR #1432 CI Verification Report

Test Results

Stateless Tests — All Passed

Integration Tests — All Passed (except 1 cancelled)

Integration Tests (amd_tsan) — 2 Failures, Unrelated to PR

Failed tests (both unrelated to PR)

Test Error Related to PR?
test_restore_db_replica/test.py::test_query_after_restore_db_replica[alter table-no exists table-no restart] Code: 60. UNKNOWN_TABLE — table does not exist during ALTER in DatabaseReplicated restore No
test_restore_db_replica/test.py::test_query_after_restore_db_replica[alter table-with exists table-no restart] Code: 60. UNKNOWN_TABLE — table does not exist during ALTER in DatabaseReplicated restore No

Both failures are in test_restore_db_replica which tests DatabaseReplicated restore operations — unrelated to Iceberg or system.tables changes. Tests are flaky.

All Iceberg tests passed (33/33)

All 33 Iceberg-related tests (test_database_iceberg, test_storage_iceberg_no_spark) in shard 4/6 passed with OK status.

Other Checks — All Passed


PR Test Coverage

PR's own integration test — Passed

The PR includes a dedicated integration test test_system_tables_partition_sorting_keys in tests/integration/test_storage_iceberg_with_spark/test_system_iceberg_metadata.py that verifies partition_key and sorting_key are correctly exposed in system.tables for Iceberg tables. This test passed in all configurations where it ran:

Test Backend Result
test_system_iceberg_metadata s3-1, s3-2, local-1, local-2, azure-1, azure-2 All OK
test_system_tables_partition_sorting_keys s3 OK

(Verified in amd_tsan shard 3/6)

clickhouse-regression dedicated test — Available

The clickhouse-regression repository has a dedicated test that independently validates this functionality:

Test: iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py

This test creates an Iceberg table with partitioning, reads it via the Iceberg table engine, and asserts that partition_key and sorting_key are correctly reported in system.tables.

@alsugiliazova alsugiliazova added the verified Approved for release label Mar 2, 2026
@vzakaznikov
Copy link
Copy Markdown
Collaborator

Audit update for PR #1432 (Iceberg partition_key / sorting_key exposure in system.tables):

Confirmed defect:

  • Medium: empty Iceberg tables can lose partition_key/sorting_key in system.tables
    • IcebergMetadata::partitionKey() and IcebergMetadata::sortingKey() both return nullopt when actual_data_snapshot is null.
    • For empty tables, getStateImpl() returns {nullptr, schema_id} when no current snapshot exists (or id < 0), even though table-level partition/sort definitions still exist in metadata.
    • Result: system.tables can show defaults/empty values for these keys on valid empty Iceberg tables.

Code anchors reviewed:

  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
  • src/Storages/System/StorageSystemTables.cpp

Suggested fix direction:

  • Derive partition/sorting display from metadata/state snapshot even when there is no data snapshot, rather than gating on actual_data_snapshot.

Regression test direction:

  • Add a system.tables assertion for an Iceberg table with partition/sort definitions before first insert/commit.

@alsugiliazova alsugiliazova added the verified-with-issues Verified by QA and issues found. label Mar 3, 2026
@alsugiliazova
Copy link
Copy Markdown
Member

Issue opened after audit #1473

zvonand added a commit that referenced this pull request Apr 16, 2026
Antalya 26.1: Expose IcebergS3 `partition_key` and `sorting_key` in `system.tables`
zvonand added a commit that referenced this pull request Apr 19, 2026
Antalya 26.1: Expose IcebergS3 `partition_key` and `sorting_key` in `system.tables`
@zvonand zvonand removed the port-antalya PRs to be ported to all new Antalya releases label Apr 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.1 antalya-26.1.6.20001 port-forward Needs to be ported to every future minor release of this major version verified Approved for release verified-with-issues Verified by QA and issues found.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

system.tables does not show partition_key/sorting_key for Iceberg table before first read/insert

5 participants