feat(general): add stats to maintenance run - CleanupMarkers #4900
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7dc6b32 to
3be87e1
Compare
3be87e1 to
d6efee9
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR adds observability statistics for the CleanupMarkers maintenance task to track the number of epoch markers and deletion watermarks removed during cleanup operations.
Key Changes:
- Introduced
CleanupMarkersStatsstruct to capture deletion counts for epoch markers and watermarks - Modified
CleanupMarkersfunction to return statistics instead of just error - Added helper functions
BuildExtraandBuildFromExtrafor serializing/deserializing stats
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| repo/maintenancestats/types.go | Implements CleanupMarkersStats type with serialization, summary generation, and builder functions |
| repo/maintenancestats/types_test.go | Adds comprehensive test coverage for stats serialization/deserialization |
| internal/epoch/epoch_manager.go | Updates CleanupMarkers and helper functions to track and return deletion counts |
| internal/epoch/epoch_manager_test.go | Updates tests to verify returned statistics |
| repo/maintenance/maintenance_run.go | Captures and propagates stats from CleanupMarkers |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
julio-lopez
requested changes
Oct 24, 2025
repo/maintenancestats/types.go
Outdated
Comment on lines
71
to
72
| DeletedEpochMarkerBlobs int `json:"deletedEpochMarkerBlobs"` | ||
| DeletedDeletionWaterMarkBlobs int `json:"deletedDeletionWaterMarkBlobs"` |
Collaborator
There was a problem hiding this comment.
Suggested change
| DeletedEpochMarkerBlobs int `json:"deletedEpochMarkerBlobs"` | |
| DeletedDeletionWaterMarkBlobs int `json:"deletedDeletionWaterMarkBlobs"` | |
| DeletedEpochMarkerBlobCount int `json:"deletedEpochMarkerBlobCount"` | |
| DeletedWatermarkBlobCount int `json:"deletedWatermarkBlobCount"` | |
Collaborator
There was a problem hiding this comment.
DeletedWatermarkBlobCount was not renamed correctly
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4900 +/- ##
==========================================
+ Coverage 75.86% 78.04% +2.17%
==========================================
Files 470 538 +68
Lines 37301 31235 -6066
==========================================
- Hits 28299 24376 -3923
+ Misses 7071 4818 -2253
- Partials 1931 2041 +110 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2b4a140 to
449932a
Compare
julio-lopez
reviewed
Oct 24, 2025
449932a to
c5d74ac
Compare
julio-lopez
reviewed
Oct 24, 2025
| // CleanupMarkersStats are the stats for cleaning up markers. | ||
| type CleanupMarkersStats struct { | ||
| DeletedEpochMarkerBlobCount int `json:"deletedEpochMarkerBlobCount"` | ||
| DeletedDeletionWaterMarkBlobCount int `json:"deletedDeletionWaterMarkBlobCount"` |
Collaborator
There was a problem hiding this comment.
From a prior comment, this can be shortened:
Suggested change
| DeletedDeletionWaterMarkBlobCount int `json:"deletedDeletionWaterMarkBlobCount"` | |
| DeletedWaterMarkBlobCount int `json:"deletedWaterMarkBlobCount"` |
Collaborator
There was a problem hiding this comment.
Let's merge this PR as is. A followup PR will address the naming of these fields.
Collaborator
There was a problem hiding this comment.
- field naming addressed in fix(general): field and variable names #4911
julio-lopez
approved these changes
Oct 24, 2025
julio-lopez
added a commit
that referenced
this pull request
Oct 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Maintenance is critical for healthy of the repository.
On the other hand, Maintenance is complex, because it runs multiple sub tasks each may generate different results according to the maintenance policy. The results may include deleting/combining/adding data/metadata to the repository.
It is worthy to add more observability for these tasks for below reasons:
There will be a serial of PRs to add observability info for each sub task.
The current PR add the stats info for CleanupMarkers sub task.