Improve reliability of large restores#4626
Conversation
Writing these blobs to their files can take a long time and consequently cause the backend connection to time out. Avoid that by retrieving these blobs separately.
a1a2ba1 to
4ea3796
Compare
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
Let's say there is a blob referenced 1000 times and has 1 MB. Does this mean there is now 1000 MB downloaded, whereas before only 1 MB was downloaded? |
|
@JsBergbau No. The amount of downloaded data hardly changes. The only difference is that frequently referenced blobs are now downloaded separately from other blobs, but still only once. For example, if |
What does this PR change? What problem does it solve?
Restoring large files can fail in some cases, for example https://forum.restic.net/t/errors-restoring-with-restic-on-windows-server-s3/6943 . This appears to be caused by blobs that are referenced a large number of times. When streaming a pack file, it takes a long time to write all instances of that blob to disk which can cause the network connection to be closed in the meantime. If writing the blob took longer than 15 minutes, then currently no retries are performed, which causes an incomplete restore.
#4624 has somewhat improved the situation by ensuring that each blob in a pack file is only processed once if streamPack has to retry the download. This helps if the blob processing delay is short enough that retries still happen, but won't fix other cases.
This PR addresses the problem from a different angle. It modifies the filerestorer such that blobs that are frequently referenced are downloaded individually and thereby avoids the timeout problem.
Was the change previously discussed in an issue or on the forum?
Related to #4605
Fixes https://forum.restic.net/t/errors-restoring-with-restic-on-windows-server-s3/6943
Checklist
[ ] I have added documentation for relevant changes (in the manual).changelog/unreleased/that describes the changes for our users (see template).gofmton the code in all commits.