Skip to content

Pruner service…#114

Merged
freemans13 merged 36 commits into
bsv-blockchain:mainfrom
freemans13:stu/cleanup-service
Nov 26, 2025
Merged

Pruner service…#114
freemans13 merged 36 commits into
bsv-blockchain:mainfrom
freemans13:stu/cleanup-service

Conversation

@freemans13

@freemans13 freemans13 commented Nov 5, 2025

Copy link
Copy Markdown
Collaborator

move preserve parent and utxo cleanup from blockassembly to cleanup service

@freemans13 freemans13 self-assigned this Nov 5, 2025
@github-actions

github-actions Bot commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

🤖 Claude Code Review

Status: Complete


Current Review

This PR successfully refactors UTXO cleanup functionality from BlockAssembly into a dedicated Pruner service. The architecture is well-designed with proper separation of concerns.

Key improvements made:

  • All previously reported issues have been resolved by the developer
  • Event-driven architecture using BlockPersisted notifications
  • Proper coordination with block persister to prevent data loss
  • Clean separation between service layer and store implementation
  • Good error handling and timeout management

No critical issues found. The code follows project conventions, includes comprehensive documentation, and handles edge cases appropriately.


History:

  • ✅ Fixed: CLI flag naming consistency (-pruner matches service name)
  • ✅ Fixed: File naming conventions (snake_case throughout)
  • ✅ Fixed: Proto directory structure (services/pruner/pruner_api/)
  • ✅ Fixed: Removed unused blobStore parameter
  • ✅ Fixed: Metrics registration follows project patterns
  • ✅ Fixed: Documentation updated to reflect 2-step pruner process

@freemans13

Copy link
Copy Markdown
Collaborator Author

🤖 Claude Code Review

Status: Complete

Current Review:

This is a well-architected refactoring that extracts cleanup logic from BlockAssembler into a dedicated Cleanup service. The design is solid with proper separation of concerns, comprehensive error handling, and good coordination patterns.

Strengths:

  • Clean extraction of cleanup responsibilities following microservices architecture
  • Proper coordination with block persister via BlockPersisted notifications to avoid data loss
  • Sequential processing with deduplication prevents flooding during block catchup
  • Three-phase cleanup (preserve parents → process expired → DAH) with abort-on-failure for safety
  • Comprehensive Prometheus metrics for observability
  • Health check implementation with dependency validation

Minor observations:

  • protobuf RPC method naming: HealthGRPC instead of Health - this appears intentional but differs from other services
  • The protobuf Go package path references bitcoin-sv/teranode (should be bsv-blockchain/teranode) but this is generated code

Overall, this refactoring improves maintainability and follows teranode's microservices patterns correctly.

  1. ✅ RPC Method Naming: HealthGRPC vs Health
    This is correct and intentional. This follows the established pattern across all teranode services:
    Health(ctx context.Context, checkLiveness bool) - Internal method used by health check framework
    HealthGRPC(ctx context.Context, *EmptyMessage) - gRPC endpoint that wraps the internal method
    Pattern confirmed in:
    services/alert/Server.go
    services/blockassembly/Server.go
    services/blockchain/Server.go
    services/blockvalidation/Server.go
    services/propagation/Server.go
    services/subtreevalidation/Server.go
    The cleanup service follows this exact pattern.
  2. ✅ Protobuf Package Path
    Already correct. The .proto file uses:
    option go_package = "github.com/bsv-blockchain/teranode/services/cleanup/cleanup_api";
    The generated .pb.go files correctly reference bsv-blockchain/teranode. There are no references to the old bitcoin-sv organization name in the cleanup service code or generated protobuf files.
    Summary
    Both observations from GitHub are non-issues:
    The HealthGRPC naming is the correct established pattern
    The protobuf package paths are already correct (bsv-blockchain/teranode)
    No changes needed! The cleanup service follows all teranode conventions correctly. ✅

Comment thread services/pruner/server.go
Comment thread services/pruner/server.go Outdated
Comment thread config/cleanup.go Outdated
Comment thread daemon/daemon_services.go Outdated
Comment thread services/cleanup/CleanupWorker.go Outdated
Comment thread services/cleanup/metrics.go Outdated
Comment thread daemon/daemon.go
Comment thread proto/cleanup/service.pb.go Outdated
@freemans13 freemans13 changed the title Cleanup service - move preserve parent and utxo cleanup from blockass… Pruner service… Nov 21, 2025
Comment thread services/pruner/pruner_worker.go Outdated
Comment thread services/pruner/server.go Outdated
Comment thread daemon/daemon.go Outdated
Comment thread services/pruner/pruner_worker.go Outdated
Comment thread services/pruner/pruner_worker.go Outdated
Comment thread services/pruner/pruner_worker.go Outdated
@freemans13

Copy link
Copy Markdown
Collaborator Author

Additional Critical Issue: CleanupWorker.go:178 references s.settings.Cleanup.JobTimeout but this field is missing from the CleanupSettings struct in config/cleanup.go. Add JobTimeout time.Duration field to fix compilation error.

Old news - sorted already

@freemans13 freemans13 requested a review from icellan November 24, 2025 14:41
Comment thread config/pruner.go Outdated
@@ -0,0 +1,22 @@
package config

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this in a new config folder? Shouldn't this just be in the service folder in the interface.go?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree - FF brain fart I didn't spot

Comment thread services/pruner/worker.go
Comment thread services/pruner/pruner_worker.go Outdated

// Safety check before DAH pruner phase
// Recheck block assembly state to ensure it hasn't changed (e.g., to reorg)
state, err = s.blockAssemblyClient.GetBlockAssemblyState(ctx)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not a duplicate of line 52?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the comment - its re-checking
I'm not convinced it should re-check anyway now I'm looking through the logic again

@freemans13 freemans13 merged commit d9ff460 into bsv-blockchain:main Nov 26, 2025
7 of 8 checks passed
torrejonv added a commit to torrejonv/teranode that referenced this pull request Dec 2, 2025
Add complete documentation for the Pruner service, a standalone
microservice extracted from Block Assembly (PR bsv-blockchain#114) responsible
for UTXO data pruning operations.

New Documentation:
- docs/topics/services/pruner.md: Main service documentation
  * Description and architecture (Container/Component diagrams)
  * Two-phase pruning process (parent preservation + DAH pruning)
  * Event-driven trigger mechanism (BlockPersisted notifications)
  * Coordination with Block Persister
  * Job queue management (LIFO pattern)
  * Migration guide from embedded cleanup
  * Configuration and deployment
- docs/references/services/pruner_reference.md: API reference
  * gRPC API (HealthGRPC endpoint)
  * Prometheus metrics (pruner_*, utxo_cleanup_*)
  * Internal interfaces
  * Event subscriptions
  * Error codes and troubleshooting
- docs/references/settings/services/pruner_settings.md: Settings reference
  * Service control (startPruner)
  * Network settings (gRPC ports and addresses)
  * Job management (pruner_jobTimeout)
  * UTXO store settings (retention, preservation, workers)
  * Context-specific configuration examples

Diagrams (PlantUML):
- C4 Container diagram (PNG)
- C4 Component diagram (PNG)
- Service initialization sequence (SVG)
- BlockPersisted trigger mechanism (SVG)
- Fallback trigger mechanism (SVG)
- Two-phase pruning process (SVG)
- Parent preservation detail (SVG)
- DAH pruning detail (SVG)
- Job queue management (LIFO) (SVG)
- Block Persister coordination (SVG)
- Detailed component diagram (SVG)

Updated Documentation:
- mkdocs.yml: Add Pruner to navigation (Topics, Reference, Settings)
- docs/references/projectStructure.md: Add Pruner service entry
- docs/topics/stores/utxo.md: Expand Data Pruning section
  * Link to Pruner service documentation
  * Explain two-phase safety mechanism
  * Document event-driven architecture

All markdown files validated with markdownlint and validate-markdown.py.
torrejonv added a commit to torrejonv/teranode that referenced this pull request Dec 2, 2025
Update Block Assembly documentation to reflect that UTXO pruning
functionality was extracted to the standalone Pruner service (PR bsv-blockchain#114).

Changes to docs/topics/services/blockAssembly.md:
- Remove section 2.7 "Unmined Transaction Cleanup" (now handled by Pruner)
- Renumber section 2.8 "Resetting the Block Assembly" to 2.7
- Update table of contents to remove cleanup section
- Replace item 9 in description: "Periodic Unmined Transaction Cleanup"
  with "Coordination with Pruner Service" and link to pruner.md

Changes to docs/references/services/blockassembly_reference.md:
- Remove cleanup-related struct fields documentation:
  * cleanupService
  * cleanupServiceLoaded
  * cleanupQueueCh
  * cleanupQueueWorkerStarted
  * unminedCleanupTicker
- Add note explaining removal in PR bsv-blockchain#114 with link to Pruner service docs
- Update lastPersistedHeight comment (remove "coordinate with cleanup")

All markdown files validated with markdownlint and validate-markdown.py.
torrejonv added a commit to torrejonv/teranode that referenced this pull request Dec 2, 2025
- Standardize "How to Run" section to match other services format
- Add comprehensive Migration Guide section (section 8) covering:
  - Architectural changes from PR bsv-blockchain#114 extraction
  - Configuration changes (new vs unchanged settings)
  - Verification steps for operators
  - Backward compatibility notes
  - Troubleshooting common migration issues
- Fix TOC link fragments to correctly reference all sections
- Remove verbose standalone execution examples in favor of simple format

The format now matches blockPersister.md and other service documentation
patterns for consistency.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants