-
Notifications
You must be signed in to change notification settings - Fork 405
[Doc](partial update) use a separate config to control the behavior of newly inserted rows in partial update #2472
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
dataroaring
merged 1 commit into
apache:master
from
bobhan1:add-partial-udpate-new-key-behavior-doc
Aug 10, 2025
Merged
[Doc](partial update) use a separate config to control the behavior of newly inserted rows in partial update #2472
dataroaring
merged 1 commit into
apache:master
from
bobhan1:add-partial-udpate-new-key-behavior-doc
Aug 10, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
befcff5 to
45de723
Compare
dataroaring
pushed a commit
to apache/doris
that referenced
this pull request
Jun 25, 2025
…f newly inserted rows in partial update (#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
a4959a3 to
5981bca
Compare
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jun 25, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jun 25, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jun 25, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jun 25, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jul 9, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jul 9, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Jul 9, 2025
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
dataroaring
approved these changes
Aug 8, 2025
Contributor
dataroaring
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
5981bca to
88fa98b
Compare
Contributor
|
run buildall |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Versions
Languages
Docs Checklist