Skip to content
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
This repository was archived by the owner on Jul 24, 2024. It is now read-only.

DBaaS v5.0.0 restore failed #1179

@apollodafoni

Description

@apollodafoni

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    If possible, provide a recipe for reproducing the error.

  2. What did you expect to see?

  3. What did you see instead?

I0607 02:17:20.643371 11 restore.go:90] [2021/06/07 02:17:20.642 +00:00] [ERROR] [import.go:267] ["download file failed"] [file="{name=1_584_32_77269f908c550594852b6adda5f616af55b7e38d973e3eb9681919c123dae892_1622622232312_write.sst,CF=write,sha256=08b8e4032cbbd54d816c37a5bcb59adb18f86894a4cd0eb5cd629b432530b4b6,startKey=?,endKey=?,startVersion=0,endVersion=425360675683958785,totalKvs=960000,totalBytes=34560000,CRC64Xor=9602817452552570455}"] [region="{ID=1756,startKey=?,endKey=?,epoch="conf_ver:5 version:327 ",peers="id:1757 store_id:1 ,id:1758 store_id:4 ,id:1759 store_id:5 "}"] [startKey=?] [endKey=?] [error="file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bb9cfa9d-9788-4c55-b5cc-a153637848f1_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d42987_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed"] [stack="github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n\tgithub.com/pingcap/br@/pkg/restore/import.go:267\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@/pkg/utils/retry.go:46\ngithub.com/pingcap/br/pkg/restore.(*FileImporter).Import\n\tgithub.com/pingcap/br@/pkg/restore/import.go:222\ngithub.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n\tgithub.com/pingcap/br@/pkg/restore/client.go:584\ngithub.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\tgithub.com/pingcap/br@/pkg/utils/worker.go:63\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
I0607 02:17:20.643438 11 restore.go:90] [2021/06/07 02:17:20.642 +00:00] [WARN] [backoff.go:77] ["unexcepted error, stop to retry"] [error="file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bb9cfa9d-9788-4c55-b5cc-a153637848f1_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d42987_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*Worke
rPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15b
c9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/bb9cfa9d-9788
-4c55-b5cc-a153637848f1_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap
/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/r
etry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgi
thub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg
/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/ping
cap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n
truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst\n git
hub.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n
tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*F
ileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n githu
b.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.c
om/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync
/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFa
iled]download sst failed\n file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downlo
adSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n
github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/b
r@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n
\tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.
com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@
v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /v
ar/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/impor
t.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgi
thub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/
br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:22
2\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGr
oup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.
go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d429
87_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*Fil
eImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n githu
b.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/
pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).Re
storeFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/wor
ker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:
1357"]
I0607 02:17:20.644808 11 restore.go:90] [2021/06/07 02:17:20.643 +00:00] [INFO] [client.go:580] ["import file done"] [file="

{name=1_584_32_77269f908c550594852b6adda5f616af55b7e38d973e3eb 9681919c123dae892_1622622232312_write.sst,CF=write,sha256=08b8e4032cbbd54d816c37a5bcb59adb18f86894a4cd0eb5cd629b432530b4b6,startKey=?,endKey=?,startVersion=0,endVersion=425360675683958785,tot alKvs=960000,totalBytes=34560000,CRC64Xor=9602817452552570455}
"] [take=54.605267947s]

  1. What version of BR and TiDB/TiKV/PD are you using?

    v5.0.0

  1. Operation logs

    • Please upload br.log for BR if possible
    • Please upload tidb-lightning.log for TiDB-Lightning if possible
    • Please upload tikv-importer.log from TiKV-Importer if possible
    • Other interesting logs
  2. Configuration of the cluster and the task

    • tidb-lightning.toml for TiDB-Lightning if possible
    • tikv-importer.toml for TiKV-Importer if possible
    • topology.yml if deployed by TiUP

dbaas使用配置参看:
https://docs.google.com/spreadsheets/d/1KFaqPTNN9BHq1sUaJOaous4DbWUo4nz5ZBDM5kQimCQ/edit#gid=0

  1. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions