branch-3.1: [opt](partial update) use a separate config to control the behavior of newly inserted rows in partial update (#41232) #52998
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
33b587b to
2963d3a
Compare
Contributor
Author
|
run buildall |
…xible partial update (apache#41950) Considering that `SegmentWriter` will be removed in the near future, we stop to support flexible partial update on `SegmentWriter` to reduce maintenance costs
…f newly inserted rows in partial update (apache#41232) Currently, Doris use strict mode to decide if newly inserted rows should be appended or report an error in partial update, which is hard to use. This PR add a new session variable and load property `partial_update_new_key_behavior` to control the behavior of newly inserted rows in partial update. `partial_update_new_key_behavior` has ~three~ two options: - `APPEND`: append the newly inserted rows - ~`IGNORE`: delete the newly inserted rows silently(will not be taken into filtered rows)~ - `ERROR`: report error if meet newly inserted rows, and the error msg will contains one row's keys which is not in table. --- The reason for not supporting `IGNORE` mode: To support `IGNORE` mode, we need to add delete sign for newly inserted rows in partial update to delete them rather than use delete bitmap mark to delete them because compaction will not use delete bitmap when reading data. Also, we need to record the rows whose delete sign is added by us in this situation for resolving conflicts in publish phase to avoid wrongly delete the rows if there are another concurrent load insert some of these rows successfully. This increases code complexity and is error-prone. Doc: apache/doris-website#2472
2963d3a to
eaf5b9d
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
211cb82 to
01c1025
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run beut |
TPC-H: Total hot run time: 39788 ms |
TPC-DS: Total hot run time: 197271 ms |
ClickBench: Total hot run time: 31.39 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
morrySnow
approved these changes
Jul 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pick #41950 and #41232