Skip to content

forget: report count of deleted files correctly#5179

Merged
MichaelEischer merged 2 commits into
restic:masterfrom
zmanda:fix-gh-5140-forget-reports-incorrect-number-of-files-deleted
Feb 2, 2025
Merged

forget: report count of deleted files correctly#5179
MichaelEischer merged 2 commits into
restic:masterfrom
zmanda:fix-gh-5140-forget-reports-incorrect-number-of-files-deleted

Conversation

@konidev20

@konidev20 konidev20 commented Dec 15, 2024

Copy link
Copy Markdown
Contributor

What does this PR change? What problem does it solve?

  • fix the issue where the progress counter would be incorrectly updated even when the file deletion failed in append-only mode discussed in gh5140
  • I have removed the progress reporter dependency from the ParallelRemove function. This will allow the function to be independently tested of the progress reporting logic.
  • I have the responsibility of progress reporting to the report function which is passed as a parameter to the ParallelRemove function. I have made the corresponding changes in the other locations.
  • Added unit tests for the changes to ensure that the report function reports the correct number of files deleted.

Was the change previously discussed in an issue or on the forum?

Closes #5140

Checklist

  • I have added tests for all code changes.
  • I have added documentation for relevant changes (in the manual).
  • There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
  • I'm done! This pull request is ready for review.

@konidev20

Copy link
Copy Markdown
Contributor Author
func ParallelRemove(ctx context.Context, repo RemoverUnpacked, fileList IDSet, fileType FileType, report func(id ID, err error) error, bar *progress.Counter) error {

Hey @MichaelEischer,

I want to remove the dependence on *progress.Counter from here, I believe the report function can update the bar as well since it's a dependency injected from outside.

This will allow restic internal methods to be tested independently of the progress reporting.

@konidev20 konidev20 marked this pull request as draft December 18, 2024 18:25
@konidev20 konidev20 marked this pull request as ready for review December 24, 2024 07:00
@konidev20 konidev20 force-pushed the fix-gh-5140-forget-reports-incorrect-number-of-files-deleted branch from 086aa5b to 75849f5 Compare December 24, 2024 10:20
@konidev20 konidev20 force-pushed the fix-gh-5140-forget-reports-incorrect-number-of-files-deleted branch 2 times, most recently from 2b992c3 to 14de45b Compare January 19, 2025 13:34

@MichaelEischer MichaelEischer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The removal of the bar.SetMax call is a significant regression and not necessary as mentioned in the below comments.

I want to remove the dependence on *progress.Counter from here, I believe the report function can update the bar as well since it's a dependency injected from outside.

This will allow restic internal methods to be tested independently of the progress reporting.

I don't understand the testing argument. In fact the test case no longer tests the behavior of ParallelRemove but rather of ParallelRemove + custom progress bar calls in report function. For consistent behavior, it feels more reasonable to have progress bar updates handled by ParallelRemove. I don't see a technical reason why that wouldn't work.

Comment thread internal/repository/prune.go
Comment thread internal/restic/parallel.go
@konidev20

Copy link
Copy Markdown
Contributor Author

I don't understand the testing argument. In fact the test case no longer tests the behavior of ParallelRemove but rather of ParallelRemove + custom progress bar calls in report function. For consistent behavior, it feels more reasonable to have progress bar updates handled by ParallelRemove. I don't see a technical reason why that wouldn't work.

I was thinking that the reporting and progress counters must be together. I can revert the change to the fix I had made earlier.

Thanks for the review. I will think about this more before making any major changes.

@konidev20 konidev20 force-pushed the fix-gh-5140-forget-reports-incorrect-number-of-files-deleted branch from 14de45b to 5e64578 Compare February 2, 2025 14:15
…orb the error

* sometimes, the report function may absorb the error and return nil, in those cases the bar.Add(1) method would execute even if the file deletion had failed
@konidev20 konidev20 force-pushed the fix-gh-5140-forget-reports-incorrect-number-of-files-deleted branch from 5e64578 to 58f58a9 Compare February 2, 2025 14:15
@MichaelEischer

Copy link
Copy Markdown
Member

I was thinking that the reporting and progress counters must be together. I can revert the change to the fix I had made earlier.

Yes, please revert the changes from 94ad25e . The implementation from 5e64578 is what I had in mind. The added tests should still work with minor modifications.

I don't think it's worth the trouble to introduce other abstractions, let's just keep using the progress counter.

@konidev20

Copy link
Copy Markdown
Contributor Author

The added tests should still work with minor modifications.

I have made the modifications, let me know if I have missed something.

@MichaelEischer MichaelEischer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

Have you seen #5140 (comment) ? Do you want to address that in this PR or should we consider this a separate issue?

@konidev20

Copy link
Copy Markdown
Contributor Author

We can address the concerns on a separate PR after discussion.

@MichaelEischer MichaelEischer merged commit efd2ec0 into restic:master Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

forget reports that files have been deleted when they can't be

2 participants