Skip to content

Add new command repoinfo#2543

Closed
aawsome wants to merge 4 commits intorestic:masterfrom
aawsome:stats-repo
Closed

Add new command repoinfo#2543
aawsome wants to merge 4 commits intorestic:masterfrom
aawsome:stats-repo

Conversation

@aawsome
Copy link
Copy Markdown
Contributor

@aawsome aawsome commented Jan 12, 2020 •
edited
Loading

What is the purpose of this change? What does it change?

Adds the possibility to show various repository statistics without walking trees, i.e. should be very fast.

The output looks like:

restic -r /path/to/repo repoinfo
enter password for repository: 
repository f1a1cc2c opened successfully, password is correct
scanning repo..

Repository content:
==================
                      count |    raw size |      crypto |   encr size
---------------------------------------------------------------------
lock files:               1 |       137 B |        32 B |       169 B
snapshot files:           1 |       222 B |        32 B |       254 B
index files:              1 |  18.270 KiB |        32 B |  18.301 KiB
data files:              39 | 190.754 MiB |         0 B | 190.754 MiB
key files:                1 |         0 B |       449 B |       449 B
---------------------------------------------------------------------
all files:               43 | 190.772 MiB |       545 B | 190.773 MiB

scanning index..

Index content:
==============
                 # packs |     # blobs |   raw blobs | pack header |      crypto |       total
----------------------------------------------------------------------------------------------
data blobs:           38 |         125 | 190.735 MiB |   4.665 KiB |   5.094 KiB | 190.744 MiB
tree blobs:            1 |           4 |   9.577 KiB |       152 B |       160 B |   9.882 KiB
----------------------------------------------------------------------------------------------
                      39 |         129 | 190.744 MiB |   4.813 KiB |   5.250 KiB | 190.754 MiB

Overhead:
=========
index:                  18.270 KiB (  0.01%)
snapshots:                   222 B (  0.00%)
locks:                       137 B (  0.00%)
pack header:             4.813 KiB (  0.00%)
crypto:                  5.782 KiB (  0.00%)
--------------------------------------------
total:                  29.082 KiB (  0.01%)

Total:
======
        129 blobs
         43 files
190.773 MiB total repository size

Was the change discussed in an issue or in the forum before?

No, but I needed some statistics while developing and thought it might be a good extension.
There is an issue with restic stat because it walks all trees for all snapshots and hence is pretty slow, see #2126. This extension does not solve this issue but is able to give repository statistics very fast.

Please give me feedback if this is useful or should be complemented by something. I will then finish this PR (and add docu, etc).
If it is not useful, feel free to close this PR.

closes #1047

Checklist

  • I have read the Contribution Guidelines
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

@rawtaz
Copy link
Copy Markdown
Contributor

rawtaz commented Feb 15, 2020

Can you replace the example of a minimal test repo with just one snapshot with a more real world example of a bigger repo?

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Feb 15, 2020

@rawtaz here is the output from a real-life repository:

Repository content:
==================
                      count |    raw size |      crypto |   encr size
---------------------------------------------------------------------
key files:                1 |         0 B |       448 B |       448 B
snapshot files:         182 |  46.834 KiB |   5.688 KiB |  52.521 KiB
index files:             59 |  32.043 MiB |   1.844 KiB |  32.044 MiB
pack files:           36299 | 170.479 GiB |         0 B | 170.479 GiB
---------------------------------------------------------------------
all files:            36541 | 170.511 GiB |   7.969 KiB | 170.511 GiB

percentage:
                      count |    raw size |      crypto |   encr size
---------------------------------------------------------------------
key files:            0.00% |       0.00% |       5.49% |       0.00%
snapshot files:       0.50% |       0.00% |      71.37% |       0.00%
index files:          0.16% |       0.02% |      23.14% |       0.02%
pack files:          99.34% |      99.98% |       0.00% |      99.98%
---------------------------------------------------------------------
all files:          100.00% |     100.00% |     100.00% |     100.00%

statistics - file size:
                        min |         max |         avg
-------------------------------------------------------
key files:            448 B |       448 B |       448 B
snapshot files:       235 B |       311 B |       295 B
index files:          240 B |   3.163 MiB | 556.159 KiB
pack files:           216 B |  11.988 MiB |   4.809 MiB
-------------------------------------------------------
all files:            216 B |  11.988 MiB |   4.778 MiB

Index content:
==============
                      count |    raw size |      crypto |   encr size
---------------------------------------------------------------------
tree blobs:           19432 |  99.245 MiB | 607.250 KiB |  99.838 MiB
data blobs:          236533 | 170.073 GiB |   7.218 MiB | 170.080 GiB
---------------------------------------------------------------------
all blobs:           255965 | 170.170 GiB |   7.811 MiB | 170.178 GiB

percentage:
                      count |    raw size |      crypto |   encr size
---------------------------------------------------------------------
tree blobs:           7.59% |       0.06% |       7.59% |       0.06%
data blobs:          92.41% |      99.94% |      92.41% |      99.94%
---------------------------------------------------------------------
all blobs:          100.00% |     100.00% |     100.00% |     100.00%

statistics - raw blobs:
                        min |         max |         avg
-------------------------------------------------------
tree blobs:            13 B | 754.551 KiB |   5.229 KiB
data blobs:             1 B |   8.000 MiB | 753.953 KiB
-------------------------------------------------------
all blobs:              1 B |   8.000 MiB | 697.112 KiB

pack size by index:
                 # packs |   raw blobs | pack header |      crypto |       total
--------------------------------------------------------------------------------
tree blobs:          229 |  99.245 MiB | 703.027 KiB | 614.406 KiB | 100.531 MiB
data blobs:        36070 | 170.073 GiB |   8.484 MiB |   8.319 MiB | 170.090 GiB
--------------------------------------------------------------------------------
all blobs:         36299 | 170.170 GiB |   9.170 MiB |   8.919 MiB | 170.188 GiB

percentage:
                 # packs |   raw blobs | pack header |      crypto |       total
--------------------------------------------------------------------------------
tree blobs:              |      98.72% |       0.68% |       0.60% |     100.00%
data blobs:              |      99.99% |       0.00% |       0.00% |     100.00%
--------------------------------------------------------------------------------
all blobs:               |      99.99% |       0.01% |       0.01% |     100.00%

Calculated packsize is 298.536 MiB smaller than the actual total size of pack files!
This means there are packs that contain blobs which are not referenced.

This is most likely due to abborted backup operations or abborted 'prune'.
It can also indicate that something is not correct with your repository.
You may consider running 'restic check' or 'restic prune'.

Overhead:
=========
index:                  32.043 MiB (  0.02%)
snapshots:              46.834 KiB (  0.00%)
pack header:             9.170 MiB (  0.01%)
crypto:                  8.927 MiB (  0.01%)
unused blobs in packs: 298.536 MiB (  0.17%)
total:                 348.722 MiB (  0.20%)

Total:
======
     255965 blobs
      36541 files
170.511 GiB total repository size

Please note that I'm using cleanup-index and cleanup-packs from #2513 and hence there exist packs that have used and unused blobs. I don't repack those packs as this is a "cold" storage where retrieving packs is much more expensive than storing some unused data.
I added reporting for this case in this PR.

greatroar
greatroar reviewed Feb 19, 2020
View reviewed changes
@aawsome aawsome mentioned this pull request Apr 19, 2020
Closed
7 tasks
@aawsome aawsome mentioned this pull request Jul 9, 2020
Closed
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Jul 9, 2020

IMO this closes #1047, but is a bit outdated.

Just need some direction: Should I rebase this to current master? Should it be changed into an restic info command?

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 2, 2020

As suggested by #1047 I changed this to implement the new command repoinfo.

@aawsome aawsome changed the title Add new mode 'repository' to restic stats Add new command repoinfo Aug 2, 2020
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 3, 2020

I overworked the implementation to make it more structured and added a few flags which enable to fine-tune the output and what repoinfo is scanning. Also set defaults which are not too verbose.

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 3, 2020

@maintainers: Actually, I would like to see this PR soon merged as this is one of my most-used commands in my own repositories (gives a good glance at your repo stats in one command) 😉

aawsome added a commit to aawsome/restic that referenced this pull request Aug 6, 2020
@aawsome
Add changelog for restic#2543
dc868c4
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 6, 2020

Added changelog and docu.

@rawtaz
Copy link
Copy Markdown
Contributor

rawtaz commented Aug 8, 2020

We'd like to think through very carefully the commands we add to the CLI, since that's one of the harder things to change in the future. Let's discuss this PR later, for now we're focusing on other work.

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 9, 2020

We'd like to think through very carefully the commands we add to the CLI, since that's one of the harder things to change in the future. Let's discuss this PR later, for now we're focusing on other work.

@rawtaz Alright, thanks for the feedback! I'll leave it in the current state with new command - at least that ensures it will stay easily mergable. Just come back to me if I need to change anything.

Maybe we should think about a possibility to include "beta"/"experimantal" functionality, i.e. functionality where the CLI is supposed to maybe change in the future? Could then be called like restic beta repoinfo...

@aawsome aawsome mentioned this pull request Aug 14, 2020
Merged
8 tasks
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 17, 2020

rebased this PR after #2885 has been merged.

@aawsome aawsome mentioned this pull request Sep 21, 2020
Closed
6 tasks
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Nov 22, 2020

I convert this to draft, as I think I should work on the following points:

Moreover the discussion about whether this is useful and whether it should be a new command or not (not to speak about a useful name) has not even started...

@aawsome aawsome marked this pull request as draft November 22, 2020 22:10
aawsome added a commit to aawsome/restic that referenced this pull request Jan 29, 2021
@aawsome
Add changelog for restic#2543
cbb045e
metalsp0rk pushed a commit to metalsp0rk/restic that referenced this pull request Dec 4, 2021
@aawsome @metalsp0rk
Add changelog for restic#2543
b5d5e1d
metalsp0rk pushed a commit to metalsp0rk/restic that referenced this pull request Feb 12, 2022
@aawsome @metalsp0rk
Add changelog for restic#2543
c81a469
metalsp0rk pushed a commit to metalsp0rk/restic that referenced this pull request Apr 25, 2022
@aawsome
Add changelog for restic#2543
1e1a2d2
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 13, 2022

I'm closing this as it seems not to be of interest for restic.

Anyone who wants to use this repoinfo command, feel free to use rustic (which offers a repoinfo command) with your restic repository.

@aawsome aawsome closed this Aug 13, 2022
@MichaelEischer MichaelEischer mentioned this pull request Jun 7, 2023
Merged
8 tasks
@aawsome aawsome deleted the stats-repo branch February 24, 2024 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@greatroar greatroar greatroar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Information command

3 participants

@aawsome @rawtaz @greatroar