List(data) failed with v0.17 on rest-server backend #4988
Description
Output of restic version
restic 0.17.0 compiled with go1.22.5 on linux/amd64
What backend/service did you use to store the repository?
rest-server
Problem description / Steps to reproduce
I am getting List(data) failed errors running check over a rest-server backend. I'm only seeing this issue on my large dataset backup (~16TB). I have other backups with sub 1TB datasets that don't have the same issue.
export RESTIC_REST_USERNAME={USER]
export RESTIC_REST_PASSWORD=[PASS]
restic -r rest:[FQDN]:8000/pool --password-file=[PASS_FILE] check
I originally opened restic/rest-server#296 on the rest-server repository. There is more data there about my specific rest-server config if that is helpful.
Expected behavior
No errors
Actual behavior
Running restic check through the rest backend was what led me to first discover this issue:
using temporary cache in /mnt/storage/tmp/cache/restic/restic-check-cache-1660270636
create exclusive lock for repository
load indexes
check all packs
List(data) returned error, retrying after 687.828349ms: List: Get "[REST]": context canceled
List(data) returned error, retrying after 1.599105373s: List: Get "[REST]": context canceled
List(data) failed: List: Get "[REST]": context canceled
List: Get "[REST]": context canceled
pack 9fad96b2d2b0c5e36e76d3e6e9bc95f7a5ee38ff587b3b5fda7848d6dc1678f0: does not exist
pack 417eeddac74cbfbb842b04b8d1946c525bc83d54e63648298849a2c099be4b63: does not exist
...
pack d9a08ec60d00b685159940bee6331c522d9b177096fb7d2cfc62e2baa9b7dc51: does not exist
pack 5454006973aeac3583d152f7f7c21cfd9641bb87718559c10dac4f775b076fb6: does not exist
pack a4d622046c6d27bcdb01e7fcbf8683347ca09456aac23ebec834e8fd0c8b9dee: does not exist
pack 7939abd14cde85bf710926ea7c7c79ac03d26f94cd9fc782637a6c6146d17e39: does not exist
pack 3ccea324c865a6bd6755157084952edc101ae2d1c7f82d0a369de3f1cfc10fdc: does not exist
pack b5b6959cc1373aff51889d2e76e4e0fb6dda670fff6ec6e9b634c0ebcfd7016e: does not exist
pack 4d0160052d10527f05ec4473415fd80b0ea9718003fac0b85f20f1a6f33561cd: does not exist
pack bf1716a2c1a3804260e75fc178644cc254d7a74b1c8b1b44122b479d2265b8b9: does not exist
pack 315dda8b42885d458dc92f3bda6bad7dd603d0e65ca98cd3535081f9dd04dc90: does not exist
check snapshots, trees and blobs
[38:17] 100.00% 11 / 11 snapshots
The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .
Fatal: repository contains errors
Running restic find on one of the 'missing' packs over rest-server:
$ restic find --pack 95f3a150d40ea4a3415b713409711aad6bc21395faa2cd653769a64d5205a4db
repository 6d3985ed opened (version 2, compression level auto)
[1:30] 100.00% 372 / 372 index files loaded
List(data) returned error, retrying after 524.900512ms: List: Get "[REST]": context canceled
List(data) returned error, retrying after 2.887577512s: List: Get "[REST]": context canceled
List(data) failed: List: Get "[REST]": context canceled
Get "[REST]": context canceled
List
github.com/restic/restic/internal/backend/rest.(*Backend).List
/restic/internal/backend/rest/rest.go:324
github.com/restic/restic/internal/backend/logger.(*Backend).List
/restic/internal/backend/logger/log.go:60
github.com/restic/restic/internal/backend/retry.(*Backend).List.func1
/restic/internal/backend/retry/backend_retry.go:258
github.com/restic/restic/internal/backend/retry.(*Backend).retry.func1
/restic/internal/backend/retry/backend_retry.go:132
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess.func1
/restic/internal/backend/retry/backend_retry.go:53
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:18
github.com/cenkalti/backoff/v4.doRetryNotify[...]
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:88
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:61
github.com/cenkalti/backoff/v4.RetryNotify
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:49
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess
/restic/internal/backend/retry/backend_retry.go:62
github.com/restic/restic/internal/backend/retry.(*Backend).retry
/restic/internal/backend/retry/backend_retry.go:130
github.com/restic/restic/internal/backend/retry.(*Backend).List
/restic/internal/backend/retry/backend_retry.go:257
github.com/restic/restic/internal/repository.(*Repository).List
/restic/internal/repository/repository.go:830
main.(*Finder).packsToBlobs
/restic/cmd/restic/cmd_find.go:412
main.runFind
/restic/cmd/restic/cmd_find.go:608
main.init.func12
/restic/cmd/restic/cmd_find.go:43
github.com/spf13/cobra.(*Command).execute
/home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
/home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
/home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext
/home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main
/restic/cmd/restic/main.go:125
runtime.main
/usr/local/go/src/runtime/proc.go:271
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
And the debug logs from rest-server while running the previous command:
HEAD /pool/config
checkConfig()
GET /pool/keys/
listBlobs()
listBlobsV2()
GET /pool/keys/ecdec7eca8457b342abc29d41b07c61d544598908085b70ccdfd228587b581b9
getBlob()
GET /pool/config
getConfig()
GET /pool/locks/
listBlobs()
listBlobsV2()
GET /pool/locks/44ec1a0b71b2d205b5a674b066f88027907518f66097eee33f60dfc1a9c8f762
getBlob()
POST /pool/locks/1fb5cc8c4cbf12d9a3012d6778e902255b72441382e7639c95dbe845222ca352
saveBlob()
GET /pool/locks/
listBlobs()
listBlobsV2()
GET /pool/locks/44ec1a0b71b2d205b5a674b066f88027907518f66097eee33f60dfc1a9c8f762
getBlob()
GET /pool/snapshots/
listBlobs()
listBlobsV2()
GET /pool/index/
listBlobs()
listBlobsV2()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/448a5b7b9742cfd64a9d77dcdbf838866675d83a78b64cec6c83abf413b481e6
saveBlob()
DELETE /pool/locks/1fb5cc8c4cbf12d9a3012d6778e902255b72441382e7639c95dbe845222ca352
deleteBlob()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/73bb041a8f900064568bc958a9fe7a3f1d6ddc39349ec80411697dd4f41ecd1a
saveBlob()
DELETE /pool/locks/448a5b7b9742cfd64a9d77dcdbf838866675d83a78b64cec6c83abf413b481e6
deleteBlob()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/d64dcc8a9f90480bcafc7a12cc407bb15de6c61484831123c556de01c591c4da
saveBlob()
DELETE /pool/locks/73bb041a8f900064568bc958a9fe7a3f1d6ddc39349ec80411697dd4f41ecd1a
deleteBlob()
DELETE /pool/locks/d64dcc8a9f90480bcafc7a12cc407bb15de6c61484831123c556de01c591c4da
deleteBlob()
I've spot checked the 'missing' packs in the data directory, and they all seem to be there. Running check locally on the server running rest-server seems to confirm this as it reports no errors.
Do you have any idea what may have caused this?
This issue started with the update to v0.17. Reverting to v0.16.5 resolves the issue.
Did restic help you today? Did it make you happy in any way?
This has been a perfect solution to my backups, and I've switch almost exclusively to restic with a combination of B2 and rest-server backends across Windows, Android, and Linux.