Note that actually new index files are checked for "fullness" and saved only after having processed a complete index. This is not totally correct and will be a problem, if this is used on one big merged index. (see #2818 )

After #2818 has been merged, I rebased this PR. Now also the newly generated index is checked for "fullness" after writing the contents of pack.

rebased after #2840 is merged.

@MichaelEischer Thanks for your feedback!
If it's ok for you, I would suggest we first focus on #2718 and deal with this PR after.
Then the changelog of #2718 can be extended here.

Also I'm a bit short on time ATM and would like to spend this making #2718 merge-ready.

Also I'm a bit short on time ATM and would like to spend this making #2718 merge-ready.

There's no need to hurry (too much). I actually hope to merge the VSS support (which shouldn't take too long anymore) before the prune rewrite PRs. That way we could release the VSS support soon (hopefully) and give the prune changes several weeks on master before getting them into a release.

Ok, I just realized that (as you can see in the failing test - one of my "edge case repos") this PR should really only be merged after #2718!

The reason is that blobs not existing in the index are processed by current prune (as the first step is to rebuild the index) but then not saved in the rebuilt index (as this only saves the contents loaded from the index files).

This situation will change with #2718 which would simply fail for this setting and would request a manual rebuild-index. So I'll build this PR on top of #2718.

I rebased this to #2718
Now there are two open things left: The test case and the progress bar.

I added the progress bar. Using packs as counter needs information from Repack how many packs have been written - which is unfortunately not available. I used blobs as counter. This information is also not available as we would need the information how many blobs have been actually written by Repack. However, if there are no duplicates or all duplicates have been repacked (one of those should be the usual case), prune knows the exact number.

I'm a bit struggling about the test case. Of course, the code is used in the integration tests and works there. As this saves the index to the repository, I cannot think about another good test except some kind of integration test. Which extra test case did you have in mind?

It would be possible to recalculate the number of packs by iterating over the index to collect all packs and then removing the rewritten ones. I haven't tested the blob counter yet, so i can't judge whether that's too noisy or not.

For the test case I have something like internal/index.TestIndexSave in mind.

It would be possible to recalculate the number of packs by iterating over the index to collect all packs and then removing the rewritten ones. I haven't tested the blob counter yet, so i can't judge whether that's too noisy or not.

I wanted to avoid the iteration and now count blobs before and after repacking to get the exact number of blobs that Repack created. This allows to exactly know how many blobs Save must save.
I don't know if I understood your "noisyness" concern correctly, but the progress bar will only update itself at most every minTickerTime (set to 1/60 s).

For the test case I have something like internal/index.TestIndexSave in mind.

I boldly copied (and adapted) that test function into internal/repositry 😉

@aawsome I've added a hook some time ago that allows injecting / wrapping a backend into OpenRepository. Search for env.gopts.backendTestHook in the integration tests.

@aawsome I've added a hook some time ago that allows injecting / wrapping a backend into OpenRepository. Search for env.gopts.backendTestHook in the integration tests.

Great, that worked for me! The test is added (also tested that the test it failes without this PR; data is listed twice) and I also mention it in the changelog.

@MichaelEischer Struggling with the test yesterday, I forgot to mention another change:
When changing the progress bar from blobs to packs, I also changed internal/index.EachByPack to yield a structure such that idx.StorePack can be used in MasterIndex.Save instead of idx.Store - this is bit faster and it doesn't fill the packIDToIndex map.

rebased this to the new version of #2718 (only docu and the PruneOptions in test was affected)

rebased after #2718 was merged.