This repository was archived by the owner on Jul 24, 2024. It is now read-only.
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
Checksum SEGFAULT when backuping datas #450
Description
Please answer these questions before submitting your issue. Thanks!
-
What did you do?
If possible, provide a recipe for reproducing the error.
I'm prepared to back up the database after I imported some data. The data was generated to reproduce the TPC-C test. It's about 1000 warehouses TPC-C test data. I use minio to act as the S3 backend. -
What did you expect to see?
Back up the database smoothly.
- What did you see instead?
I ran the command as below:
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
./tidb-toolkit-nightly-linux-amd64/bin/br backup full \
--pd "172.24.57.167:2379" \
--storage "s3://mybucket/backups?endpoint=http://172.24.57.167:9000" \
--send-credentials-to-tikv true \
--log-file backuptable.log \
--checksum=trueWhen I'm backing up, the minio reported as below:
API: PutObjectPart(bucket=mybucket, object=backups/2_2120_67_6f9a767ce2abddef675b3e5005301427bad8cbc4c619c8acf2b2f2f0b112e2f2_default.sst)
Time: 14:44:17 CST 08/06/2020
DeploymentID: 6a84a6d5-8f6b-4733-9c07-0c3884d819c4
RequestID: 16289AC95C253524
RemoteHost: 172.24.57.168
Host: 172.24.57.167:9000
UserAgent: rusoto/0.43.0 rust/1.42.0-nightly linux
Error: file not found
uploadID=81109bc3-c347-4c54-aad5-8d198111cca3, uploadIDDir=/root/backups/.minio.sys/multipart/27f1fd6e2e6ec05bef047f9e2badd34e0f32e891c65fa55f86973038d248dec4/81109bc3-c347-4c54-aad5-8d198111cca3
1: github.com/minio/minio@/cmd/fs-v1-multipart.go:91:cmd.(*FSObjects).backgroundAppend()
API: PutObjectPart(bucket=mybucket, object=backups/1_1964_58_a4afb9bb968858c2afff47b60ac5e07a395a146616ae198dd3e283c633b8e1af_default.sst)
Time: 14:44:17 CST 08/06/2020
DeploymentID: 6a84a6d5-8f6b-4733-9c07-0c3884d819c4
RequestID: 16289AC95D23C050
RemoteHost: 172.24.57.166
Host: 172.24.57.167:9000
UserAgent: rusoto/0.43.0 rust/1.42.0-nightly linux
Error: file not found
uploadID=7b711b6d-5bb3-48eb-80ae-f7dee52af30f, uploadIDDir=/root/backups/.minio.sys/multipart/3bb1657fce68d01000fa46429dc3ceb27bc95327c07089c38780f1652ef1145e/7b711b6d-5bb3-48eb-80ae-f7dee52af30f
1: github.com/minio/minio@/cmd/fs-v1-multipart.go:91:cmd.(*FSObjects).backgroundAppend()
API: PutObjectPart(bucket=mybucket, object=backups/2_3900_89_ad17bb537cf96804513fb2db42d81a2861ab98f5e70318eadbf44e7fabd4baaa_write.sst)
Time: 14:45:46 CST 08/06/2020
DeploymentID: 6a84a6d5-8f6b-4733-9c07-0c3884d819c4
RequestID: 16289ADDE96668F3
RemoteHost: 172.24.57.168
Host: 172.24.57.167:9000
UserAgent: rusoto/0.43.0 rust/1.42.0-nightly linux
Error: file not found
uploadID=03e707d5-cae3-4408-b780-750004a7b12e, uploadIDDir=/root/backups/.minio.sys/multipart/c2bd0d9b1a648cfa9f3b9fd7cf13ac3ef6ec0d8300289ca9c88a5fb80b302502/03e707d5-cae3-4408-b780-750004a7b12e
1: github.com/minio/minio@/cmd/fs-v1-multipart.go:91:cmd.(*FSObjects).backgroundAppend()
When processing in checksum stage, BR reported as below:
Detail BR log in backuptable.log
Full backup <----------------------------------------------------------------------------------------------------------> 100.00%Checksum <------------------------------------|.........................................................................> 33.33%panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1fb973c]
goroutine 412 [running]:
github.com/pingcap/tidb/store/tikv.(*copIterator).recvFromRespCh(0xc0011740a0, 0x3477d80, 0xc0006aa5c0, 0xc00134fec0, 0x0, 0xc000920000)
/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200805053026-cd3e5ed82671/store/tikv/coprocessor.go:591 +0x1cc
github.com/pingcap/tidb/store/tikv.(*copIterator).Next(0xc0011740a0, 0x3477d80, 0xc0006aa5c0, 0xc00093da50, 0x107def8, 0x30, 0x2d75240)
/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200805053026-cd3e5ed82671/store/tikv/coprocessor.go:640 +0x225
github.com/pingcap/tidb/distsql.(*selectResult).NextRaw(0xc000c702a0, 0x3477d80, 0xc0006aa5c0, 0xc00102a420, 0xc000b741b0, 0x0,
0x347a500, 0xc000c702a0)
/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200805053026-cd3e5ed82671/distsql/select_result.go:173 +0x4c
github.com/pingcap/br/pkg/checksum.sendChecksumRequest(0x3477d80, 0xc0006aa5c0, 0x344bc00, 0xc00102a420, 0xc000b741b0, 0xc001076f30, 0x0, 0x0)
/home/jenkins/agent/workspace/build_br_multi_branch_master/go/src/github.com/pingcap/br/pkg/checksum/executor.go:221 +0x1cf
github.com/pingcap/br/pkg/checksum.(*Executor).Execute(0xc0010368e0, 0x3477d80, 0xc0006aa5c0, 0x344bc00, 0xc00102a420, 0x3035178, 0xc00102a420, 0xc000e17c18, 0x1fc2b03)
/home/jenkins/agent/workspace/build_br_multi_branch_master/go/src/github.com/pingcap/br/pkg/checksum/executor.go:276 +0xed
github.com/pingcap/br/pkg/backup.calculateChecksum(0x3477d80, 0xc0006aa5c0, 0xc00078d080, 0x344bc00, 0xc00102a420, 0x5cf0a0cf4600002, 0x0, 0x0, 0x0)
/home/jenkins/agent/workspace/build_br_multi_branch_master/go/src/github.com/pingcap/br/pkg/backup/schema.go:152 +0xbe
github.com/pingcap/br/pkg/backup.(*Schemas).Start.func1.1(0xc000801700, 0x11028dc)
/home/jenkins/agent/workspace/build_br_multi_branch_master/go/src/github.com/pingcap/br/pkg/backup/schema.go:80 +0x191
github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1(0x0, 0x0)
/home/jenkins/agent/workspace/build_br_multi_branch_master/go/src/github.com/pingcap/br/pkg/utils/worker.go:62 +0x74
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001054660, 0xc0010646c0)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:54 +0x66
- What version of BR and TiDB/TiKV/PD are you using?
Release Version: v4.0.0-beta.2-93-g181c081
Git Commit Hash: 181c081
Git Branch: master
Go Version: go1.13
UTC Build Time: 2020-08-05 12:12:01
Race Enabled: false
TiDB,TiKV,PD v4.0.4