Skip to content

Add support for shared storage snapshots across subqueries to ensure query-level consistency#79471

Merged
azat merged 18 commits intoClickHouse:masterfrom
amosbird:projection-index-3
Jun 9, 2025
Merged

Add support for shared storage snapshots across subqueries to ensure query-level consistency#79471
azat merged 18 commits intoClickHouse:masterfrom
amosbird:projection-index-3

Conversation

@amosbird
Copy link
Copy Markdown
Collaborator

@amosbird amosbird commented Apr 23, 2025

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added a setting enable_shared_storage_snapshot_in_query to enable sharing the same storage snapshot across all subqueries in a single query. This ensures consistent reads from the same table, even when the table is referenced multiple times within a query.

For example, the following query now produces correct and stable results when this setting is enabled:

SELECT count()
FROM events
WHERE _part_starting_offset + _part_offset IN (
    SELECT _part_starting_offset + _part_offset
    FROM events
    WHERE user_id = 42
)
SETTINGS enable_shared_storage_snapshot_in_query = 1

This helps #78429.

Additionally, introduced a test-only setting merge_tree_storage_snapshot_sleep_ms that allows injecting a delay during storage snapshot acquisition, to simulate concurrent modifications and verify snapshot consistency.

Note: This feature currently only applies to tables in the MergeTree family.

Caveat: Enabling this feature disables the merge_tree_enable_remove_parts_from_snapshot_optimization, which may lead to old data parts being retained longer during long-running queries. See #37913.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@amosbird amosbird changed the title Projection index 3 Add support for shared storage snapshots across subqueries to ensure query-level consistency Apr 23, 2025
@amosbird amosbird force-pushed the projection-index-3 branch from fee46e3 to 9b19f76 Compare April 23, 2025 14:12
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Apr 23, 2025

Workflow [PR], commit [e53c668]

@clickhouse-gh clickhouse-gh bot added the pr-feature Pull request with new product feature label Apr 23, 2025
Copy link
Copy Markdown
Member

@azat azat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK to me

@KochetovNicolai can you also PTAL?

@azat azat self-assigned this Apr 23, 2025
@amosbird amosbird force-pushed the projection-index-3 branch 3 times, most recently from c829d84 to 577278d Compare April 29, 2025 14:20
Copy link
Copy Markdown
Member

@azat azat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've discussed this with @KochetovNicolai and he is OK with it

BTW @amosbird I guess we can cache parts after partition pruning for this case?

P.S. can you please resolve the conflicts (and sorry for the delay)

@amosbird
Copy link
Copy Markdown
Collaborator Author

I guess we can cache parts after partition pruning for this case?

We can't, because each subquery may have different pruning predicates.

P.S. can you please resolve the conflicts (and sorry for the delay)

Sure!

@amosbird amosbird force-pushed the projection-index-3 branch from 577278d to 4cd0f8e Compare May 22, 2025 12:36
@amosbird
Copy link
Copy Markdown
Collaborator Author

Stateless tests (coverage, 6/6) — fail: 1, passed: 1301, skipped: 9

03519_storage_url #80591 (comment)

@azat azat enabled auto-merge May 25, 2025 21:31
@azat azat disabled auto-merge May 25, 2025 21:33
@azat
Copy link
Copy Markdown
Member

azat commented May 25, 2025

CH Inc syncWaiting for status to be reported — Conflicts found. The sync PR is not mergeable, it probably has conflicts

@amosbird I've fixed the conflicts with private, so please don't rebase from now on

@azat
Copy link
Copy Markdown
Member

azat commented May 26, 2025

CH Inc sync — tests failed

03443_shared_storage_snapshots is flaky in private - apparently 1.5 second is not enough to ensure that the snapshot won't be shared, i.e. the query might not be started in this 1.5 second, let's try to increase the timeouts (merge_tree_storage_snapshot_sleep_ms=10000 and sleep 5)

If it will fail again we can use failpoints instead.

@azat azat added this pull request to the merge queue Jun 9, 2025
Merged via the queue into ClickHouse:master with commit 0c4e588 Jun 9, 2025
120 checks passed
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 9, 2025
baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jun 10, 2025
baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jun 10, 2025
baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jun 10, 2025
baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jun 11, 2025
baibaichen pushed a commit to apache/gluten that referenced this pull request Jun 11, 2025
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250611)

* Fix Build due to ClickHouse/ClickHouse#81191

* Fix UT due to ClickHouse/ClickHouse#79471

---------

Co-authored-by: kyligence-git <gluten@kyligence.io>
Co-authored-by: Chang chen <chenchang@apache.com>
@rschu1ze
Copy link
Copy Markdown
Member

@amosbird @azat enable_shared_storage_snapshot_in_query seems crucial from a correctness perspective. Why is this not enabled by default?

@amosbird
Copy link
Copy Markdown
Collaborator Author

@rschu1ze It’s already in progress: #82634. I believe @azat has shared some comments there about the potential downsides.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants