Skip to content

S3: Don't archive metadata files on S3 Glacier #4584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MichaelEischer merged 2 commits into from
Jan 20, 2024
Merged

S3: Don't archive metadata files on S3 Glacier #4584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8ca58b4
S3: do not set storage class for metadata when using archive storage
elkemper Dec 4, 2023
a763a5c
s3: minor cleanups for archive storage class handling
MichaelEischer Jan 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions changelog/unreleased/issue-4583
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Enhancement: Ignore s3.storage-class for metadata if archive tier is specified

There is no official cold storage support in restic, use this option at your
own risk.

Restic always stored all files on s3 using the specified `s3.storage-class`.
Now, restic will store metadata using a non-archive storage tier to avoid
problems when accessing a repository. To restore any data, it is still
necessary to manually warm up the required data beforehand.

https://github.com/restic/restic/issues/4583
https://github.com/restic/restic/pull/4584
25 changes: 19 additions & 6 deletions internal/backend/s3/s3.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,16 +325,29 @@ func (be *Backend) Path() string {
return be.cfg.Prefix
}

// useStorageClass returns whether file should be saved in the provided Storage Class
// For archive storage classes, only data files are stored using that class; metadata
// must remain instantly accessible.
func (be *Backend) useStorageClass(h backend.Handle) bool {
notArchiveClass := be.cfg.StorageClass != "GLACIER" && be.cfg.StorageClass != "DEEP_ARCHIVE"
isDataFile := h.Type == backend.PackFile && !h.IsMetadata
return isDataFile || notArchiveClass
}

// Save stores data in the backend at the handle.
func (be *Backend) Save(ctx context.Context, h backend.Handle, rd backend.RewindReader) error {
objName := be.Filename(h)

opts := minio.PutObjectOptions{StorageClass: be.cfg.StorageClass}
opts.ContentType = "application/octet-stream"
// the only option with the high-level api is to let the library handle the checksum computation
opts.SendContentMd5 = true
// only use multipart uploads for very large files
opts.PartSize = 200 * 1024 * 1024
opts := minio.PutObjectOptions{
ContentType: "application/octet-stream",
// the only option with the high-level api is to let the library handle the checksum computation
SendContentMd5: true,
// only use multipart uploads for very large files
PartSize: 200 * 1024 * 1024,
}
if be.useStorageClass(h) {
opts.StorageClass = be.cfg.StorageClass
}

info, err := be.client.PutObject(ctx, be.cfg.Bucket, objName, io.NopCloser(rd), int64(rd.Length()), opts)

Expand Down