Store snapshot statistics & print snapshot size by MichaelEischer · Pull Request #4705 · restic/restic

MichaelEischer · 2024-02-22T21:50:51Z

What does this PR change? What problem does it solve?

Store the backup statistics in a snapshot. Sample output:

{
  "time": "2024-02-22T22:39:19.991058932+01:00",
  "parent": "2c29113d80ef2f4072bae3095963a9c70f9d7014e893afd74887c8b61d30f143",
  "tree": "c163608e60d5bf1921b296528cd28b9844b508d96d9f4ad2564ced61c6dd0a24",
  "paths": [
    "/home/michael/Projekte/restic/restic"
  ],
  "hostname": "host",
  "username": "michael",
  "uid": 1000,
  "gid": 1000,
  "program_version": "restic 0.16.4-dev (compiled manually)",
  "summary": {
    "backup_start": "2024-02-22T22:39:19.991058932+01:00",
    "backup_end": "2024-02-22T22:39:26.37292521+01:00",
    "files_new": 85,
    "files_changed": 155,
    "files_unmodified": 2493,
    "dirs_new": 3,
    "dirs_changed": 110,
    "dirs_unmodified": 328,
    "data_blobs": 234,
    "tree_blobs": 114,
    "data_added": 39375231,
    "data_added_in_repo": 19236421,
    "total_files_processed": 2733,
    "total_bytes_processed": 152165603
  }
}

In addition the snapshots command now also prints the snapshot size (based on total_bytes_processed).

ID        Time                 Host        Tags        Paths                                    Size
-----------------------------------------------------------------------------------------------------------
4e0ff0ef  2024-02-22 22:39:19  host                    /home/michael/Projekte/restic/restic     145.116 MiB
-----------------------------------------------------------------------------------------------------------

Was the change previously discussed in an issue or on the forum?

Fixes #693
Fixes #874

Checklist

I have read the contribution guidelines.
I have enabled maintainer edits.
I have added tests for all code changes.
I have added documentation for relevant changes (in the manual).
There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
I have run gofmt on the code in all commits.
All commit messages are formatted in the same style as the other commits in the repo.
I'm done! This pull request is ready for review.

darkdragon-001 · 2024-02-22T22:19:23Z

Does the --json output also contain the size?

aawsome · 2024-02-23T07:17:22Z

@MichaelEischer Is there a reason why your implementation deviates from
#693 (comment) ?

MichaelEischer · 2024-02-23T18:33:16Z

Differences to #693 (comment):

"data_added_in_repo" instead of "data_added_packed": the corresponding fields in ItemStats are called DataSizeInRepo and TreeSizeInRepo. Thus that name is more consistent.
"data_added_files", "data_added_files_packed", "data_added_trees", "data_added_trees_packed": Not available in the JSON output
"total_dirs_processed", "total_dirsize_processed": these statistics are not tracked so far
"total_duration", "backup_duration": both values feel pretty redundant to me
"command": I'm somewhat undecided whether it's a good idea or might store unwanted details

Although, the more precise description would be that the summary contains the fields from json.summaryOutput minus SnapshotID, DryRun, TotalDuration and MessageType, and adds BackupStart and BackupEnd.

I thought about omitting BackupStart and BackupEnd altogether and just adding TotalDuration. But as it's possible to pass --time to the backup command, the time of a snapshot and BackupStart can differ.

Does the --json output also contain the size?

Yes, it even includes the full statistics information stored in the snapshot.

aawsome · 2024-02-23T18:58:58Z

Differences to #693 (comment):

* "data_added_in_repo" instead of "data_added_packed": the corresponding fields in `ItemStats` are called `DataSizeInRepo` and `TreeSizeInRepo`. Thus that name is more consistent.

If you name them like this you make the snapshot format incompatible to what rustic implemented some 2 years ago. I don't care too much if you omit the other fields, but same things should be named the same, IMO.

The value describes how much data was added after compression. Previously, it was only available in the text output.

hraban · 2024-02-24T02:22:49Z

Just for the record this is only meaningful until you start deleting other snapshots which share data, right?

aawsome · 2024-02-24T04:24:34Z

Just for the record this is only meaningful until you start deleting other snapshots which share data, right?

No. It is as meaningful as it is saving the output of backup --json in some database. If you are trying to derive "how much size does this snapshot uniquely occupy in the repo" from the information, you are making a mistake as this is not what it states. This information cannot be derived even after a follow-up backup run (which may share data).
The information is only a "snapshot" of what happened during the backup run. Including information about the source (which could also change directly after the backup run). But for analysis reasons the information might be interesting.

hraban · 2024-02-24T16:46:54Z

No. It is as meaningful as it is saving the output of backup --json in some database. If you are trying to derive "how much size does this snapshot uniquely occupy in the repo" from the information, you are making a mistake as this is not what it states. This information cannot be derived even after a follow-up backup run (which may share data). The information is only a "snapshot" of what happened during the backup run. Including information about the source (which could also change directly after the backup run). But for analysis reasons the information might be interesting.

Right, that's what it seemed like. It's useful in the moment, but if you have e.g. auto cleanup of older backups it quickly becomes stale. Can still be useful of course, but not the same as e.g. TimeMachine's "calculatedrift" option (which would presumably be a lot more expensive to run).

MichaelEischer · 2024-02-25T19:42:43Z

I've renamed the attribute to data_added_packed. The text output uses that value in %s stored which might also have served as a suffix. But that's just bikeshedding, I'm fine with either variant. So let's use the one that maintains compatibility.

In general, we're interested in a friendly coexistence with rustic. But it should also be clear that we won't always agree how things should be implemented. Nevertheless there's still value in trying to maintain compatibility where it makes sense. That judgement will still be up to us, but that doesn't mean the we do X just because rustic implemented Y.

Is there a reason why your implementation deviates from #693 (comment) ?

Let me add just one thing here: the PR was intentionally still marked as draft as the naming issue was not yet resolved.

aawsome · 2024-02-27T09:39:04Z

I've renamed the attribute to data_added_packed.

Thanks for changing this. I think this helps user who use both restic and rustic. And it's about our users what we are doing, right?
I will change rustic such that it can work with omitting the fields you don't fill in restic.

In general, we're interested in a friendly coexistence with rustic.

Good to hear. I am also very interested in a friendly coexistence. This is all free open source software!

MichaelEischer

LGTM.

mirabilos · 2024-03-28T23:45:01Z

changelog/unreleased/issue-693

+created using this or a future restic version. For this, the `backup` command
+now stores the backup summary statistics in the snapshot.
+
+The text output of the `snapshots` command only shows the snapshot size. The


What exactly of all the discussed options is the “snapshot size” here?

Bytes added to the repo when this backup was created?

If so, fine with me, but I would still love to also see the “amount of data backed up” (what others call restore size) for a quick double-check.

It is total_bytes_processed which IMO should be the restore size.

MichaelEischer mentioned this pull request Feb 22, 2024

Print out the backup size when listing snapshots (enhancement) #693

Closed

MichaelEischer added 6 commits February 23, 2024 20:24

archiver: unexport save/saveDir/saveTree methods

8b1a857

archiver: refactor summary collection from ui into the archiver

a59f654

backup: expose data_added_in_repo in JSON output

8689731

The value describes how much data was added after compression. Previously, it was only available in the text output.

backup: store statistics in snapshot

38f91d3

snapshots: Print snapshot size stored in snapshots

b652003

archiver: test backup summary calculation

6813959

MichaelEischer force-pushed the snapshot-statistics branch from add8958 to 6813959 Compare February 23, 2024 20:47

MichaelEischer added 2 commits February 23, 2024 22:05

archiver: test statistics in snapshot

a8f5684

document snapshot statistics

6a13e45

This comment was marked as off-topic.

Sign in to view

backup: rename data_added_in_repo statistic to data_added_packed

e71660c

This comment was marked as off-topic.

Sign in to view

MichaelEischer marked this pull request as ready for review February 25, 2024 19:53

MichaelEischer mentioned this pull request Feb 25, 2024

Make restore resumable #3425

Closed

8 tasks

This comment was marked as off-topic.

Sign in to view

MichaelEischer mentioned this pull request Mar 28, 2024

Feature Request: include 'bytes added to repository' in progress line while backing up #4736

Open

MichaelEischer commented Mar 28, 2024

View reviewed changes

MichaelEischer merged commit 7f9ad1c into restic:master Mar 28, 2024

MichaelEischer deleted the snapshot-statistics branch March 28, 2024 21:41

mirabilos reviewed Mar 28, 2024

View reviewed changes

MichaelEischer mentioned this pull request May 25, 2024

More stats in snapshots list #874

Closed

BrewTestBot mentioned this pull request Jul 26, 2024

restic 0.17.0 Homebrew/homebrew-core#178554

Merged

k0ste mentioned this pull request Mar 11, 2025

[feat] support for restic snapshots metadata ngosang/restic-exporter#38

Closed

Conversation

MichaelEischer commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR change? What problem does it solve?

Was the change previously discussed in an issue or on the forum?

Checklist

Uh oh!

darkdragon-001 commented Feb 22, 2024

Uh oh!

aawsome commented Feb 23, 2024

Uh oh!

MichaelEischer commented Feb 23, 2024

Uh oh!

aawsome commented Feb 23, 2024

Uh oh!

hraban commented Feb 24, 2024

Uh oh!

aawsome commented Feb 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hraban commented Feb 24, 2024

Uh oh!

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

MichaelEischer commented Feb 25, 2024

Uh oh!

This comment was marked as off-topic.

This comment was marked as off-topic.

aawsome commented Feb 27, 2024

Uh oh!

This comment was marked as off-topic.

This comment was marked as off-topic.

MichaelEischer left a comment

Choose a reason for hiding this comment

Uh oh!

mirabilos Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

aawsome Mar 29, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MichaelEischer commented Feb 22, 2024 •

edited

Loading

aawsome commented Feb 24, 2024 •

edited

Loading