Decode error when "NULL" value in the column with "primary key" flag

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)



TiFlash assumes all values in the column with "primary key" flag are non-null value. https://github.com/pingcap/tiflash/blob/6d0cbc8cc78b4048fb88e0845d6c9c2881e01515/dbms/src/Storages/Transaction/RowCodec.cpp#L323-L330

However, users could create such "NULL" value in the "primary key" column by following steps

```sql
-- create table with `source` is nullable
create table test.t_case (
`case_no` varchar(32) not null,
`source` varchar(20) default null,
`p` varchar(12) DEFAULT NULL,
primary key (`case_no`)
);

-- insert some data and left `source` to be empty
insert into test.t_case(case_no)
values ("1"), ("2"), ("3"), ("4");

-- drop the primary key, fill the `source` to be non-empty
alter table t_case drop primary key;
update test.t_case set `source` = '' where `source` is NULL;
-- add new primary key with case_no and source
alter table t_case add primary key (case_no, source);

-- send the snapshot data to tiflash
alter table t_case set tiflash replica 1;
```

### 2. What did you expect to see? (Required)

No exception thrown

### 3. What did you see instead (Required)

Exception throw with "Parameter out of bound in IColumnString::insertRangeFrom method" in v4.0.x/v5.0.x/v5.1.x/v5.2.x/v5.3.x/v5.4.0~v5.4.1/v6.1.0

```
[2022/09/13 15:30:02.786 +08:00] [ERROR] [<unknown>] ["bool DB::GCManager::work():Code: 12, e.displayText() = DB::Exception: Parameter out of bound in IColumnString::insertRangeFrom method, start=0, length=8, src_size=4:  while reading DTFile /data2/jaysonhuang/tiup-deploy/data/tiflash-5018/data/t_77/stable/dmf_2: while apply gc Segment [1] [range=[-9223372036854775808,9223372036854775807)] [table=t_77], e.what() = DB::Exception, Stack trace:
0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x36d42e6]
1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x36c8116]
2. bin/tiflash/tiflash(DB::ColumnString::insertRangeFrom(DB::IColumn const&, unsigned long, unsigned long)+0x2ff) [0x759712f]
3. bin/tiflash/tiflash(DB::DM::copyColumnsData(std::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > > const&, COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x7fe) [0x76f690e]
4. bin/tiflash/tiflash(DB::DM::DPFileReader::readRowsOnce(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x135) [0x77059c5]
5. bin/tiflash/tiflash(DB::DM::DPFileReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x55) [0x7706d85]
6. bin/tiflash/tiflash(DB::DM::DeltaValueReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0xcf) [0x771c34f]
7. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, false>::writeInsertFromDelta(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long&)+0x94) [0x7698344]
8. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, false>::read()+0x863) [0x769b593]
9. bin/tiflash/tiflash(DB::DM::DMRowKeyFilterBlockInputStream<true>::read()+0x5f) [0x7692d2f]
10. bin/tiflash/tiflash(DB::DM::PKSquashingBlockInputStream<false>::read()+0x9f1) [0x7698d61]
11. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock()+0x2f) [0x7a3dcbf]
12. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x1dd0) [0x7a411a0]
13. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read()+0x18) [0x7681968]
14. bin/tiflash/tiflash(DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DM::DMFileWriter::Flags)+0x20b) [0x7670cdb]
15. bin/tiflash/tiflash(DB::DM::createNewStable(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::IBlockInputStream> const&, unsigned long, DB::DM::WriteBatches&)+0xbd) [0x767143d]
16. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0x17d) [0x767ab7d]
17. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3e1) [0x76496b1]
18. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::onSyncGc(long)+0x5c1) [0x7650601]
19. bin/tiflash/tiflash(DB::GCManager::work()+0x7b1) [0x7a09911]
20. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x947) [0x75dba17]
21. bin/tiflash/tiflash() [0x8b14f6f]
22. /lib64/libpthread.so.0(+0x6e63) [0x7fe4a07f6e63]
23. /lib64/libc.so.6(clone+0x6d) [0x7fe4a022b55d]
"] [thread_id=10]
```


Exception message after v5.4.2/v6.1.1/v6.2.0
```
[2022/09/13 15:59:30.623 +08:00] [ERROR] [Exception.cpp:85] ["DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t):Code: 9, e.displayText() = DB::Exception: Sizes of columns doesn't match: _INTERNAL_DELMARK(id=-1025): 8, source(id=2): 4, e.what() = DB::Exception, Stack trace:
       0x1cdb0ce    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30257358]
                    dbms/src/Common/Exception.h:41
       0x65b93c4    DB::Block::checkNumberOfRows() const [tiflash+106664900]
                    dbms/src/Core/Block.cpp:243
       0x735076e    bool DB::RegionBlockReader::readImpl<(DB::TMTPKType)0>(DB::Block&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > > const&, bool) [tiflash+120915822]
                    dbms/src/Storages/Transaction/RegionBlockReader.cpp:211
       0x732cc13    DB::GenRegionBlockDataWithSchema(std::__1::shared_ptr<DB::Region> const&, std::__1::shared_ptr<DB::DecodingStorageSchemaSnapshot const> const&, unsigned long, bool, DB::TMTContext&) [tiflash+120769555]
                    dbms/src/Storages/Transaction/PartitionStreams.cpp:606
       0x72ee5a6    DB::DM::SSTFilesToBlockInputStream::readCommitedBlock() [tiflash+120513958]
                    dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:238
       0x72edcca    DB::DM::SSTFilesToBlockInputStream::read() [tiflash+120511690]
                    dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:144
       0x71d25c5    DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+119350725]
                    dbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:253
       0x72eeec2    DB::DM::PKSquashingBlockInputStream<true>::read() [tiflash+120516290]
                    dbms/src/Storages/DeltaMerge/PKSquashingBlockInputStream.h:68
       0x71d25c5    DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+119350725]
                    dbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:253
       0x1c9f065    DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock() [tiflash+30011493]
                    dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:134
       0x1c9d16c    DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+30003564]
                    dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:51
       0x72e954c    DB::DM::SSTFilesToDTFilesOutputStream::write() [tiflash+120493388]
                    dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:162
       0x72e2dc3    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+120466883]
                    dbms/src/Storages/Transaction/ApplySnapshot.cpp:340
       0x733183a    PreHandleSnapshot [tiflash+120789050]
                    dbms/src/Storages/Transaction/ProxyFFI.cpp:387
  0x7f059e8b3e2a    raftstore::engine_store_ffi::_$LT$impl$u20$raftstore..engine_store_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h718eff40e0ff22c6 [libtiflash_proxy.so+33197610]
  0x7f059e86be89    raftstore::store::snap::Snapshot::pre_handle_snapshot::h81c04c547dfcb1a8 [libtiflash_proxy.so+32902793]
  0x7f059df01536    yatp::task::future::RawTask$LT$F$GT$::poll::h68cf0c4ec7b4731e [libtiflash_proxy.so+23029046]
  0x7f059f51ec0d    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h6502e9a2b4ccd66e [libtiflash_proxy.so+46218253]
  0x7f059f52143d    std::sys_common::backtrace::__rust_begin_short_backtrace::hc5d6c9c6e9339eea [libtiflash_proxy.so+46228541]
  0x7f059f51413d    core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h8a6505e26346d93a [libtiflash_proxy.so+46174525]
  0x7f059ec5ca0a    std::sys::unix::thread::Thread::new::thread_start::hd39c5f08bdcda277 [libtiflash_proxy.so+37034506]
  0x7f059bfece63    start_thread [libpthread.so.0+28259]
                    /root/yuzhao/glibc-2.17/nptl/pthread_create.c:308
  0x7f059bb0655d    __clone [libc.so.6+968029]
                    /root/yuzhao/glibc-2.17/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113"] [thread_id=48]
```
### 4. What is your TiFlash version? (Required)
v5.4.1

	inline bool addDefaultValueToColumnIfPossible(const ColumnInfo & column_info, Block & block, size_t block_column_pos, bool force_decode)
	{
	// We consider a missing column could be safely filled with NULL, unless it has not default value and is NOT NULL.
	// This could saves lots of unnecessary schema syncs for old data with a schema that has newly added columns.
	// for clustered index, if the pk column does not exists, it can still be decoded from the key
	if (column_info.hasPriKeyFlag())
	return true;

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decode error when "NULL" value in the column with "primary key" flag #5859

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Decode error when "NULL" value in the column with "primary key" flag #5859

Description

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions