feat(general): add stats to maintenance run - ExtendBlobRetentionStats #4956

Lyndon-Li · 2025-11-05T10:59:59Z

Maintenance is critical for healthy of the repository.
On the other hand, Maintenance is complex, because it runs multiple sub tasks each may generate different results according to the maintenance policy. The results may include deleting/combining/adding data/metadata to the repository.

It is worthy to add more observability for these tasks for below reasons:

It is helpful for troubleshooting. Any data change to the repository is critical, the observability info helps to understand what happened during the maintenance and why that happened
It is helpful for users to understand/predict the repo's behavior. The repo data may be stored in a public cloud for which costs are sensitive to scale/duration of data stored. On the other hand, repository has its own policy to manage the data, so the data is not deleted until it is safe enough according to the policy. The observability info helps users to understand how much data is in-use, how much data is out of use and when it is deleted

There will be a serial of PRs to add observability info for each sub task.
The current PR add the stats info for ExtendBlobRetention sub task.

Lyndon-Li · 2025-11-05T11:05:48Z

repo/maintenance/blob_retain.go

 // ExtendBlobRetentionTime extends the retention time of all relevant blobs managed by storage engine with Object Locking enabled.
-func ExtendBlobRetentionTime(ctx context.Context, rep repo.DirectRepositoryWriter, opt ExtendBlobRetentionTimeOptions) (int, error) {
+//
+//nolint:funlen


The length of this function(108) has exceeded the limit (100 lines).
I didn't find a way to split it without breaking its struct and logic.

So nolint:funlen is added for now.

To be addressed in a separate PR.

Copilot

Pull Request Overview

This PR adds observability for the ExtendBlobRetention maintenance task by introducing structured statistics tracking. The enhancement allows users to monitor blob retention extension operations, which is valuable for troubleshooting maintenance activities and understanding repository behavior in cloud storage environments where costs are sensitive to data retention periods.

Key Changes:

Introduces ExtendBlobRetentionStats struct to track blob retention extension metrics (blobs to extend, blobs extended, retention period)
Updates ExtendBlobRetentionTime function to return statistics instead of just a count
Adds comprehensive test coverage for the new statistics type

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
repo/maintenancestats/stats_extend_blob_retention.go	Defines the new `ExtendBlobRetentionStats` struct with JSON serialization and human-readable summary methods
repo/maintenancestats/builder.go	Registers the new stats type in the builder's switch statement for deserialization
repo/maintenancestats/builder_test.go	Adds test cases for serialization/deserialization of `ExtendBlobRetentionStats`
repo/maintenance/blob_retain.go	Refactors `ExtendBlobRetentionTime` to create and return `ExtendBlobRetentionStats` instead of integer count
repo/maintenance/maintenance_run.go	Updates task runner to properly handle the new stats return value
repo/maintenance/blob_retain_test.go	Updates existing tests to validate the returned statistics

Copilot · 2025-11-05T20:51:50Z

repo/maintenance/blob_retain_test.go

+	stats, err := maintenance.ExtendBlobRetentionTime(ctx, env.RepositoryWriter, maintenance.ExtendBlobRetentionTimeOptions{})
 	require.NoError(t, err)
+	require.Equal(t, uint32(4), stats.BlobsExtended)
+	require.Equal(t, uint32(4), stats.BlobsExtended)


Line 77 duplicates the assertion from line 76. This appears to be a copy-paste error. The second assertion should verify stats.BlobsToExtend instead of stats.BlobsExtended again.

Suggested change

require.Equal(t, uint32(4), stats.BlobsExtended)

require.Equal(t, uint32(4), stats.BlobsToExtend)

Merging as is, fixing in a separate PR.

addressed in refactor(general): fix names in ExtendBlobRetentionStats #4958

codecov · 2025-11-05T21:02:41Z

Codecov Report

❌ Patch coverage is 73.07692% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.02%. Comparing base (cb455c6) to head (b952213).
⚠️ Report is 733 commits behind head on master.

Files with missing lines	Patch %	Lines
repo/maintenance/blob_retain.go	69.23%	4 Missing ⚠️
...po/maintenancestats/stats_extend_blob_retention.go	80.00%	2 Missing ⚠️
repo/maintenance/maintenance_run.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4956      +/-   ##
==========================================
+ Coverage   75.86%   78.02%   +2.15%     
==========================================
  Files         470      545      +75     
  Lines       37301    31368    -5933     
==========================================
- Hits        28299    24475    -3824     
+ Misses       7071     4848    -2223     
- Partials     1931     2045     +114

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

julio-lopez · 2025-11-05T22:29:44Z

repo/maintenance/blob_retain_test.go

+	stats, err := maintenance.ExtendBlobRetentionTime(ctx, env.RepositoryWriter, maintenance.ExtendBlobRetentionTimeOptions{})
 	require.NoError(t, err)
+	require.Equal(t, uint32(4), stats.BlobsExtended)
+	require.Equal(t, uint32(4), stats.BlobsExtended)


Merging as is, fixing in a separate PR.

julio-lopez · 2025-11-05T22:30:57Z

repo/maintenancestats/stats_extend_blob_retention.go

+	BlobsToExtend   uint32 `json:"blobsToExtend"`
+	BlobsExtended   uint32 `json:"blobsExtended"`


These are "blob counts", right ?

Suggested change

BlobsToExtend uint32 `json:"blobsToExtend"`

BlobsExtended uint32 `json:"blobsExtended"`

ToExtendBlobCount uint32 `json:"toExtendBlobCount"`

ExtendedBlobCount uint32 `json:"extendedBlobCount"`

Fixing in refactor(general): fix names in ExtendBlobRetentionStats #4958

julio-lopez

#4848

julio-lopez · 2025-11-06T00:11:51Z

repo/maintenancestats/stats_extend_blob_retention.go

+	BlobsToExtend   uint32 `json:"blobsToExtend"`
+	BlobsExtended   uint32 `json:"blobsExtended"`


Fixing in refactor(general): fix names in ExtendBlobRetentionStats #4958

julio-lopez · 2025-11-06T00:12:24Z

repo/maintenance/blob_retain.go

 // ExtendBlobRetentionTime extends the retention time of all relevant blobs managed by storage engine with Object Locking enabled.
-func ExtendBlobRetentionTime(ctx context.Context, rep repo.DirectRepositoryWriter, opt ExtendBlobRetentionTimeOptions) (int, error) {
+//
+//nolint:funlen


To be addressed in a separate PR.

julio-lopez · 2025-11-06T00:12:43Z

repo/maintenance/blob_retain_test.go

+	stats, err := maintenance.ExtendBlobRetentionTime(ctx, env.RepositoryWriter, maintenance.ExtendBlobRetentionTimeOptions{})
 	require.NoError(t, err)
+	require.Equal(t, uint32(4), stats.BlobsExtended)
+	require.Equal(t, uint32(4), stats.BlobsExtended)


addressed in refactor(general): fix names in ExtendBlobRetentionStats #4958

blob retention extention stats

b952213

Lyndon-Li commented Nov 5, 2025

View reviewed changes

Lyndon-Li marked this pull request as ready for review November 5, 2025 11:06

julio-lopez requested review from Rohit-BM18, andrason and Copilot November 5, 2025 20:51

Copilot AI reviewed Nov 5, 2025

View reviewed changes

julio-lopez reviewed Nov 5, 2025

View reviewed changes

julio-lopez mentioned this pull request Nov 6, 2025

refactor(general): fix names in ExtendBlobRetentionStats #4958

Merged

julio-lopez approved these changes Nov 6, 2025

View reviewed changes

julio-lopez merged commit af38dfc into kopia:master Nov 6, 2025
22 of 23 checks passed

julio-lopez mentioned this pull request Nov 6, 2025

refactor(general): extendBlobRetentionTime #4960

Merged

	require.Equal(t, uint32(4), stats.BlobsExtended)
	require.Equal(t, uint32(4), stats.BlobsToExtend)

		BlobsToExtend uint32 `json:"blobsToExtend"`
		BlobsExtended uint32 `json:"blobsExtended"`

feat(general): add stats to maintenance run - ExtendBlobRetentionStats #4956

feat(general): add stats to maintenance run - ExtendBlobRetentionStats #4956

Uh oh!

Conversation

Lyndon-Li commented Nov 5, 2025

Uh oh!

Lyndon-Li Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julio-lopez Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julio-lopez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Lyndon-Li Nov 5, 2025 •

edited

Loading

codecov bot commented Nov 5, 2025 •

edited

Loading

julio-lopez Nov 5, 2025 •

edited

Loading