S3: Don't archive metadata files on S3 Glacier#4584
S3: Don't archive metadata files on S3 Glacier#4584MichaelEischer merged 2 commits intorestic:masterfrom
Conversation
MichaelEischer
left a comment
There was a problem hiding this comment.
I don't understand how this PR is supposed to work. The directory metadata of a snapshot is also stored in pack files and consequently there's no chance for the backend to distinguish data and metadata.
|
Hey @MichaelEischer, thanks for picking the PR! Here's my line of thought when I investigated this issue. Well, after your question, I've done some more digging and went up on call stack and found that it used in all Also, as a proof of work - here's bunch of screenshots of S3 when I tried this code. (saving just one small text file) And in opposite, here's my first attempts of saving to the Glacier, when I didn't know about issue. Hope this helps. |
|
A pack file can contain either tree or data blobs. To allow a second host without a prefilled cache to use the repository for backups, the second host must also be able to retrieve the tree blobs. However, both tree and data blobs are stored in pack files. To see what I mean just try the following (while using your PR): The second backup run will use the snapshot from the first backup run as its parent snapshot. And it will try to download the corresponding tree blobs, which will fail. For that to work a pack file must also not be archived if I'm not entirely sure how you intend the |
changelog/unreleased/issue-4583
Outdated
| thus snapshot metadata could be accessed from machine without local cache. | ||
|
|
||
| https://github.com/restic/restic/issues/4583 | ||
| https://github.com/restic/restic/issues/3202 No newline at end of file |
There was a problem hiding this comment.
This PR does not add full blown support for archive storage. As such it should be clear after reading that there is no official support for cold storages.
|
Thanks for the comments. As of how I'm expecting to use this backups - to make it the "1" from "3-2-1" backup strategy. I want a cheap storage for valuable media only in case of failures of local copies and raid setup. So yeah, if I will restore it, it will be a full restore. |
6bc929a to
ffbed06
Compare
MichaelEischer
left a comment
There was a problem hiding this comment.
LGTM.
I've reworded the changelog and made a few additional minor cleanups.
ffbed06 to
a763a5c
Compare









What does this PR change? What problem does it solve?
It fixes issue when you upload snapshot to S3 Glacier, and metadata isn't available from any machine without cache.
It sets a fallback for the case where storage class should not be set - for metadata AND ONLY when specified tier will move files to archive. For the code simplicity it is done through the inverse logic operators: set storage class when file is
data/OR when storage class is not instant accessible (such as Standard etc. including 'Glacier Instant Retrieval').Was the change previously discussed in an issue or on the forum?
Closes #4583
Checklist
changelog/unreleased/that describes the changes for our users (see template).gofmton the code in all commits.