Fix decode error when "NULL" value in the column with "primary key" flag (#5879) by ti-chi-bot · Pull Request #5933 · pingcap/tiflash

ti-chi-bot · 2022-09-16T13:44:14Z

This is an automated cherry-pick of #5879

What problem does this PR solve?

Issue Number: close #5859

Problem Summary:
TiFlash assumes all values in the column with "primary key" flag are non-null value.

However, users could create such "NULL" value in the "primary key" column by following steps

-- create table with `source` is nullable
create table test.t_case (
`case_no` varchar(32) not null,
`source` varchar(20) default null,
`p` varchar(12) DEFAULT NULL,
primary key (`case_no`)
);

-- insert some data and left `source` to be empty
insert into test.t_case(case_no)
values ("1"), ("2"), ("3"), ("4");

-- drop the primary key, fill the `source` to be non-empty
alter table t_case drop primary key;
update test.t_case set `source` = '' where `source` is NULL;
-- add new primary key with case_no and source
alter table t_case add primary key (case_no, source);

-- send the snapshot data to tiflash
alter table t_case set tiflash replica 1;

What is changed and how it works?

mainly change the logic in addDefaultValueToColumnIfPossible
- For non-clustered index table(is_common_handle=false && pk_is_handle=false), fill default value even if the column contains primary key flag when the column is not appear in the encoded value(those rows are not supposed to be return to users)
- For clustered index/pk_is_handle table, ignore the primary key column as they are supposed to be decoded from the encoded key
Refine the exception message in insertRangeFrom
Add more region info when exception thrown in preHandleSnapshot/ingestSST

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fix the panic issue after creating the primary index with a column containing `NULL` value

ti-chi-bot · 2022-09-16T13:44:15Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

JaySon-Huang · 2022-09-20T10:29:28Z

/rebuild

JaySon-Huang · 2022-09-23T06:26:36Z

/rebuild

JaySon-Huang · 2022-09-23T06:52:50Z

/rebuild

JaySon-Huang · 2022-09-23T09:04:14Z

/run-all-tests

…lag (pingcap#5879) close pingcap#5859

Signed-off-by: JaySon-Huang <tshent@qq.com>

JaySon-Huang · 2022-09-26T08:14:09Z

/run-all-tests

JaySon-Huang · 2022-09-26T08:16:36Z

/label can-merge

ti-chi-bot · 2022-09-26T08:16:38Z

@JaySon-Huang: The label(s) can-merge cannot be applied. These labels are supported: tide/merge-method-rebase, tide/merge-method-squash, tide/merge-method-merge, duplicate, first-time-contributor, good first issue, help wanted, invalid, question, wontfix, make-local-great-again, needs-cherry-pick-release-3.1, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0-rc, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, needs-rebase.

Details

In response to this:

/label can-merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JaySon-Huang · 2022-09-26T08:17:11Z

/label status/can-merge

ti-chi-bot · 2022-09-26T08:17:12Z

@JaySon-Huang: The label(s) status/can-merge cannot be applied. These labels are supported: tide/merge-method-rebase, tide/merge-method-squash, tide/merge-method-merge, duplicate, first-time-contributor, good first issue, help wanted, invalid, question, wontfix, make-local-great-again, needs-cherry-pick-release-3.1, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0-rc, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, needs-rebase.

Details

In response to this:

/label status/can-merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JaySon-Huang · 2022-09-26T08:22:36Z

/run-unit-test

JaySon-Huang · 2022-09-26T08:31:14Z

/run-unit-test

Signed-off-by: JaySon-Huang <tshent@qq.com>

JaySon-Huang · 2022-09-26T10:16:06Z

/merge

ti-chi-bot · 2022-09-26T10:16:07Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-26T10:16:08Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 48994e2

sre-bot · 2022-09-26T10:34:30Z

Coverage for changed files

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Columns/ColumnAggregateFunction.cpp                              121               121     0.00%          31                31     0.00%         262               262     0.00%          72                72     0.00%
Columns/ColumnArray.cpp                                          418               418     0.00%          45                45     0.00%         652               652     0.00%         260               260     0.00%
Columns/ColumnDecimal.cpp                                        115                83    27.83%          17                13    23.53%         243               173    28.81%          62                48    22.58%
Columns/ColumnFixedString.cpp                                    120               117     2.50%          18                17     5.56%         223               216     3.14%          72                71     1.39%
Columns/ColumnString.cpp                                         140                85    39.29%          16                10    37.50%         264               161    39.02%          82                52    36.59%
Columns/ColumnVector.cpp                                         114                58    49.12%          21                11    47.62%         212               106    50.00%          68                36    47.06%
Debug/dbgFuncMockRaftSnapshot.cpp                                174               162     6.90%          23                15    34.78%         553               505     8.68%         118               116     1.69%
Storages/DeltaMerge/SSTFilesToBlockInputStream.h                   4                 4     0.00%           4                 4     0.00%           4                 4     0.00%           0                 0         -
Storages/Transaction/ApplySnapshot.cpp                           342               161    52.92%          13                 4    69.23%         500               245    51.00%         166               104    37.35%
Storages/Transaction/PartitionStreams.cpp                        262               213    18.70%          21                13    38.10%         563               389    30.91%         138               117    15.22%
Storages/Transaction/RegionBlockReader.cpp                        77                25    67.53%           3                 0   100.00%         156                24    84.62%          56                23    58.93%
Storages/Transaction/RegionBlockReader.h                          26                26     0.00%          10                10     0.00%          34                34     0.00%          20                20     0.00%
Storages/Transaction/RowCodec.cpp                                252                29    88.49%          18                 0   100.00%         405                55    86.42%         214                40    81.31%
Storages/Transaction/tests/RowCodecTestUtils.h                    81                 3    96.30%          14                 0   100.00%         171                 0   100.00%          40                 5    87.50%
Storages/Transaction/tests/gtest_kvstore.cpp                     270                15    94.44%          23                 2    91.30%        1223                32    97.38%          10                 6    40.00%
Storages/Transaction/tests/gtest_region_block_reader.cpp         266                60    77.44%          29                 1    96.55%         424                13    96.93%          74                27    63.51%
Storages/Transaction/tests/region_helper.h                         4                 0   100.00%           4                 0   100.00%          29                 0   100.00%           0                 0         -
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                           2786              1580    43.29%         310               176    43.23%        5918              2871    51.49%        1452               997    31.34%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18366      9786             46.72%    206542  97986        52.56%

full coverage report (for internal network access only)

ti-chi-bot added do-not-merge/cherry-pick-not-approved release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 16, 2022

ti-chi-bot mentioned this pull request Sep 16, 2022

Fix decode error when "NULL" value in the column with "primary key" flag #5879

Merged

12 tasks

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-6.1 This PR is cherry-picked to release-6.1 from a source PR. labels Sep 16, 2022

ti-chi-bot assigned JaySon-Huang Sep 16, 2022

JaySon-Huang force-pushed the cherry-pick-5879-to-release-6.1 branch from 006d886 to 9dd8cb7 Compare September 20, 2022 06:10

VelocityLight added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Sep 26, 2022

lidezhu approved these changes Sep 26, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 26, 2022

JaySon-Huang added 3 commits September 26, 2022 15:48

Fix decode error when "NULL" value in the column with "primary key" f…

ad34bd9

…lag (pingcap#5879) close pingcap#5859

fix include

623cc42

Signed-off-by: JaySon-Huang <tshent@qq.com>

Try fix compile error

07d1f2e

Signed-off-by: JaySon-Huang <tshent@qq.com>

JaySon-Huang force-pushed the cherry-pick-5879-to-release-6.1 branch from 41ec07a to 07d1f2e Compare September 26, 2022 07:55

Fix fullstack test

9d6b6f4

Signed-off-by: JaySon-Huang <tshent@qq.com>

JaySon-Huang approved these changes Sep 26, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 26, 2022

Fix ut

48994e2

Signed-off-by: JaySon-Huang <tshent@qq.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 26, 2022

ti-chi-bot merged commit 443e921 into pingcap:release-6.1 Sep 26, 2022

Conversation

ti-chi-bot commented Sep 16, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Sep 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaySon-Huang commented Sep 20, 2022

Uh oh!

JaySon-Huang commented Sep 23, 2022

Uh oh!

JaySon-Huang commented Sep 23, 2022

Uh oh!

JaySon-Huang commented Sep 23, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

ti-chi-bot commented Sep 26, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

ti-chi-bot commented Sep 26, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

JaySon-Huang commented Sep 26, 2022

Uh oh!

ti-chi-bot commented Sep 26, 2022

Uh oh!

ti-chi-bot commented Sep 26, 2022

Uh oh!

sre-bot commented Sep 26, 2022

Coverage for changed files

Coverage summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ti-chi-bot commented Sep 16, 2022 •

edited

Loading