Intermittently fails to remove Backblaze locks #2562
Description
Output of restic version
restic 0.9.5 compiled with go1.11.5 on linux/amd64
How did you run restic exactly?
export RESTIC_PASSWORD="xxx"
export B2_ACCOUNT_ID="xxx"
export B2_ACCOUNT_KEY="xxx"
/usr/bin/restic --quiet --repo "b2:<bucket>:/" backup "<path>"
What backend/server/service did you use to store the repository?
Backblaze
Expected behavior
Silence. Complete backup. No locks left around.
Actual behavior
First time this happen we got:
Remove(<lock/e2da98e949>) returned error, retrying after 372.362036ms: Delete: b2_authorize_account: 401:
Remove(<lock/e2da98e949>) returned error, retrying after 943.465489ms: Delete: b2_download_file_by_name: 404: bucket xxx does not have file: locks/e2da98e9499d5675c4e2dcb02712fea406b19913933fc849c23e6d0f59797b49
Remove(<lock/e2da98e949>) returned error, retrying after 1.47376298s: Delete: b2_download_file_by_name: 404: bucket xxx does not have file: locks/e2da98e9499d5675c4e2dcb02712fea406b19913933fc849c23e6d0f59797b49
Remove(<lock/e2da98e949>) returned error, retrying after 1.711066009s: Delete: b2_download_file_by_name: 404: bucket xxx does not have file: locks/e2da98e9499d5675c4e2dcb02712fea406b19913933fc849c23e6d0f59797b49
Remove(<lock/e2da98e949>) returned error, retrying after 1.735833725s: Delete: b2_download_file_by_name: 404: bucket xxx does not have file: locks/e2da98e9499d5675c4e2dcb02712fea406b19913933fc849c23e6d0f59797b49
...
After which the lock was wedged and we had to manually unlock to make things work again.
Second time just happened, with this error:
Remove(<lock/3a93d0ba6c>) returned error, retrying after 722.19857ms: Delete: b2_delete_file_version: 401:
Fatal: unable to save snapshot: Close: b2_finish_large_file: 400: No active upload for: 4_z517d80021a42b51268e10718_f206d9e98194c45df_d20200128_m142022_c000_v0001065_t0056
And there is no snapshot present after this run.
Steps to reproduce the behavior
No idea what causes this. It was ten days between the current incidents.
Do you have any idea what may have caused this?
Nope.
Do you have an idea how to solve the issue?
Nope.
Did restic help you today? Did it make you happy in any way?
Not today as the backup failed. But otherwise. :)