add docs about insertion deduplication on retries#2394
Merged
justindeguzman merged 13 commits intomainfrom Jun 28, 2024
Merged
Conversation
kssenii
reviewed
Jun 26, 2024
| That transformed data also has to be deduplicated on retries. Clickhouse deduplicates it in the same way it deduplicates data which is inserted into target table. | ||
| User could control that process with settings on the table under materialized view: `replicated_deduplication_window`, `replicated_deduplication_window_seconds` and `non_replicated_deduplication_window`. Also user could use profile setting `deduplicate_blocks_in_dependent_materialized_views`. | ||
| For the blocks inserted in tables under materialized views Clickhouse calulates `block_id` as hash from a string which contains concatenation of `block_id`'s for the the source table and other parts which helps to distiguish blocks afret materialised views transformation, like source view's id and the sequential number of that block. That makes deduplication for materialised views works correctly and distinguish the data by its original inserted data, no mater how it has been transormed on its way to the destination table under materialized view. | ||
|
|
Member
There was a problem hiding this comment.
Also mention that update_insert_deduplication_token_in_dependent_materialized_views is deprecated after some version?
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
justindeguzman
approved these changes
Jun 28, 2024
Member
Author
|
Did this docs go to the production? The feature has not been merged yet... |
Contributor
|
Sorry about that, will revert asap. I'll just hide it from the sidebar and it won't be accessible. I thought you wanted me to edit it and then merge. |
Member
Author
|
Yes, I want that. I should have filled the description for that PR as well, sorry. |
18 tasks
Member
Author
Did you reverted it? |
Contributor
|
Yes this was removed |
Member
Author
|
The code is committed, I resurrect this doc in a day or two. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.