Overview

This document outlines the architectural change to remove the transaction distribution functionality from the RPC service. The Distributor component will be moved to a separate library, and the RPC sendrawtransaction command will be simplified to only submit transactions to the cluster's propagation service(s).

Current Architecture

Current Flow

Client → RPC sendrawtransaction → Distributor → Multiple Propagation Services
                                  ├─ Retry logic
                                  ├─ Failure tolerance
                                  ├─ Load balancing
                                  └─ Multi-server distribution

Current Components

Distributor (services/rpc/Distributor.go)

• Purpose: Reliable transaction propagation to multiple propagation service instances
• Features:
  • Multi-server transaction distribution with automatic failover
  • Configurable retry logic with exponential backoff
  • Failure tolerance allowing partial success scenarios
  • Performance monitoring and response time tracking
  • Concurrent transaction submission for improved throughput
  • HTTP client connection pooling and reuse

Configuration Settings

Located in settings/settings.go:

• distributor_backoff_duration - Backoff between retries (default: 1s)
• distributor_max_retries - Maximum retry attempts (default: 3)
• distributor_failure_tolerance - Percentage of failures allowed (default: 0%)
• distributer_wait_time - Wait time between transactions (default: 0)
• distributor_timeout - Timeout for each distribution attempt (default: 30s)

Current Usage in RPC Service

File: services/rpc/handlers.go:742-755

d, err := NewDistributor(context.Background(), s.logger, s.settings)
if err != nil {
    return nil, errors.NewServiceError("could not create distributor", err)
}

res, err := d.SendTransaction(context.Background(), tx)
if err != nil {
    return nil, &bsvjson.RPCError{
        Code:    bsvjson.ErrRPCInvalidParameter,
        Message: "TX rejected: " + err.Error(),
    }
}

Proposed Architecture

New Flow

Client → RPC sendrawtransaction → Propagation Service(s)
                                 └─ Simple gRPC call(s)

Client → External Library → Distributor → Multiple Nodes/Services
         (if distribution needed)

Design Principles

1. Separation of Concerns: RPC service should only handle the JSON-RPC protocol and forward transactions to the local cluster's propagation service
2. Single Responsibility: Distribution logic belongs in a separate library that clients can use if needed
3. Simplicity: Core Teranode RPC should not include cross-node distribution logic
4. Client Control: Clients requiring distribution should control that logic themselves

Implementation Steps

Phase 1: Create External Distribution Library

Task 1.1: Create new library package

• Create new repository/package: teranode-tx-distributor
• Move Distributor.go to new package
• Move Distributor_test.go to new package
• Move Distributor_comprehensive_test.go to new package
• Update package declaration and imports

Task 1.2: Update library interfaces

• Make Distributor a standalone component
• Add constructor that accepts propagation service addresses
• Ensure library can work independently of Teranode settings
• Add configuration struct for distributor options
• Document library API and usage examples

Task 1.3: Publish library

• Add proper README with usage examples
• Add LICENSE file
• Create initial release/tag
• Publish to appropriate package registry

Phase 2: Simplify RPC sendrawtransaction Handler

Task 2.1: Update handler implementation

File: services/rpc/handlers.go

Current location: Lines 742-755

Changes needed:

// OLD CODE (remove):
d, err := NewDistributor(context.Background(), s.logger, s.settings)
if err != nil {
    return nil, errors.NewServiceError("could not create distributor", err)
}

res, err := d.SendTransaction(context.Background(), tx)
if err != nil {
    return nil, &bsvjson.RPCError{
        Code:    bsvjson.ErrRPCInvalidParameter,
        Message: "TX rejected: " + err.Error(),
    }
}

// NEW CODE (replace with):
// Get propagation client from server
if s.propagationClient == nil {
    return nil, &bsvjson.RPCError{
        Code:    bsvjson.ErrRPCInternal,
        Message: "propagation client not available",
    }
}

// Send transaction directly to cluster's propagation service
err = s.propagationClient.ProcessTransaction(ctx, tx)
if err != nil {
    return nil, &bsvjson.RPCError{
        Code:    bsvjson.ErrRPCInvalidParameter,
        Message: "TX rejected: " + err.Error(),
    }
}

// Return transaction hash
return tx.TxIDChainHash().String(), nil

Task 2.2: Add propagation client to RPCServer

File: services/rpc/Server.go

Add field to RPCServer struct (around line 687):

// propagationClient provides access to the propagation service
// Used for submitting transactions to the cluster
propagationClient *propagation.Client

Task 2.3: Initialize propagation client

Update NewServer function in services/rpc/Server.go:

func NewServer(logger ulogger.Logger, tSettings *settings.Settings,
    blockchainClient blockchain.ClientI, blockValidationClient blockvalidation.Interface,
    utxoStore utxo.Store, blockAssemblyClient blockassembly.ClientI,
    peerClient peer.ClientI, p2pClient p2p.ClientI) (*RPCServer, error) {

    // ... existing code ...

    // Create propagation client
    propagationAddresses := tSettings.Propagation.GRPCAddresses
    if len(propagationAddresses) == 0 {
        return nil, errors.NewConfigurationError("no propagation service addresses configured")
    }

    // Use first propagation address (or implement simple round-robin if needed)
    pConn, err := util.GetGRPCClient(context.Background(), propagationAddresses[0],
        &util.ConnectionOptions{MaxRetries: 3}, tSettings)
    if err != nil {
        return nil, errors.NewServiceError("failed to connect to propagation service", err)
    }

    propagationClient, err := propagation.NewClient(context.Background(), logger, tSettings, pConn)
    if err != nil {
        return nil, errors.NewServiceError("failed to create propagation client", err)
    }

    rpc := RPCServer{
        // ... existing fields ...
        propagationClient: propagationClient,
    }

    // ... rest of function ...
}

Phase 3: Remove Distributor from RPC Service

Task 3.1: Remove Distributor files

• Delete services/rpc/Distributor.go
• Delete services/rpc/Distributor_test.go
• Delete services/rpc/Distributor_comprehensive_test.go

Task 3.2: Remove configuration settings

File: settings/settings.go

Remove settings (around lines 406-410):

• Remove DistributorBackoffDuration
• Remove DistributorMaxRetries
• Remove DistributorFailureTolerance
• Remove DistributerWaitTime
• Remove DistributorTimeout

File: settings/interface.go

Remove corresponding fields from Coinbase struct

File: settings.conf

Remove corresponding configuration entries

Task 3.3: Update imports

Review and update imports in:

• services/rpc/handlers.go
• services/rpc/Server.go

Phase 4: Update Tests

Task 4.1: Update RPC tests

Files to update:

• services/rpc/handlers_test.go
• services/rpc/handlers_additional_test.go
• services/rpc/server_test.go

Changes needed:

• Remove distributor-related mocks
• Add propagation client mocks
• Update test expectations to match new simple flow
• Ensure sendrawtransaction tests verify direct propagation calls

Task 4.2: Update integration tests

Files that use Distributor (from grep results):

• test/utils/transaction_helper.go
• test/utils/testenv.go
• test/utils/helper.go
• test/tnj/locktime_test.go
• test/tnb/**/*_test.go
• test/e2e/**/*_test.go

Changes needed:

• Update transaction submission helpers
• Remove distributor initialization
• Update tests to work with direct propagation
• Add tests using external library if needed

Task 4.3: Update mock files

File: services/rpc/mock.go

• Remove distributor-related mocks
• Add propagation client mocks if needed

Phase 5: Update Documentation

Task 5.1: Update service documentation

File: docs/topics/services/rpc.md

• Remove distributor references
• Update architecture diagrams
• Document new simplified flow
• Add section on when/how to use external distribution library

Task 5.2: Update PlantUML diagrams

Files:

• docs/topics/services/img/plantuml/rpc/rpc_detailed_component.puml
• docs/topics/services/img/plantuml/rpc/RPC_Component.svg
• docs/topics/services/img/plantuml/rpc/rpc-send-raw-transaction.puml
• docs/topics/services/img/plantuml/rpc/rpc-send-raw-transaction.svg

Changes:

• Remove Distributor component
• Show direct connection to Propagation service
• Regenerate SVG files from updated .puml files

Task 5.3: Update how-to guides

File: docs/howto/submitting_transactions.md

• Update examples to show new API
• Add section on using external distribution library
• Update code examples

Task 5.4: Update Server.go documentation

File: services/rpc/Server.go

• Update package-level documentation (lines 1-42)
• Remove references to Distributor component
• Update architecture description

Task 5.5: Create migration guide

Create new file: docs/migration/rpc-distributor-removal.md

• Document breaking changes
• Provide migration examples
• Link to external distribution library
• Explain new responsibilities for clients

Phase 6: Update Dependencies

Task 6.1: Update go.mod

• Add external distribution library dependency (if clients need it)
• Run go mod tidy

Task 6.2: Update vendor

• Run go mod vendor if using vendoring

Phase 7: Update CI/CD

Task 7.1: Update CI tests

• Ensure all tests pass with new implementation
• Update any CI scripts that reference distributor settings
• Update deployment configurations

Task 7.2: Update Docker configurations

• Remove distributor settings from docker-compose files
• Update environment variable documentation

Migration Guide for Clients

Before (Current Implementation)

// Client sends transaction via RPC
// RPC service handles distribution to multiple nodes automatically
result := rpcClient.SendRawTransaction(txHex)

After (New Implementation)

Option 1: Simple single-cluster submission

// Client sends transaction via RPC
// RPC service sends to local cluster's propagation service only
result := rpcClient.SendRawTransaction(txHex)

Option 2: Multi-node distribution (if needed)

import "github.com/bsv-blockchain/teranode-tx-distributor"

// Client handles distribution to multiple nodes themselves
distributor := txdistributor.New(
    txdistributor.WithServers([]string{
        "node1.example.com:8081",
        "node2.example.com:8081",
        "node3.example.com:8081",
    }),
    txdistributor.WithRetries(3),
    txdistributor.WithBackoff(1 * time.Second),
    txdistributor.WithFailureTolerance(33), // 33% can fail
)

responses, err := distributor.DistributeTransaction(ctx, tx)

Benefits of This Change

1. Clearer Separation of Concerns: RPC service focuses on JSON-RPC protocol, not distribution logic
2. Reduced Complexity: RPC service has fewer dependencies and configuration options
3. Better Testability: Simpler RPC handler is easier to test
4. Flexibility: Clients can choose distribution strategy based on their needs
5. Maintainability: Distribution logic in separate library can evolve independently
6. Performance: Removes unnecessary overhead for clients that don't need distribution
7. Scalability: Clients can implement custom distribution strategies

Risks and Considerations

Breaking Changes

• Existing clients relying on automatic distribution will need to update
• Configuration settings will be removed (breaking for deployments)
• API response format may change (currently returns array of responses, will return single response)

Backward Compatibility

• Consider providing compatibility shim for one release cycle
• Document migration path clearly
• Provide code examples for common use cases

Performance Impact

• Clients needing distribution will have additional round-trip latency
• However, most clients don't need distribution and will benefit from reduced latency

Testing Requirements

• Comprehensive integration tests for new flow
• Performance benchmarks to ensure no regression
• E2E tests for various client scenarios

Timeline Estimate

• Phase 1: Create External Library - 1-2 weeks
• Phase 2: Simplify RPC Handler - 3-5 days
• Phase 3: Remove Distributor - 2-3 days
• Phase 4: Update Tests - 1-2 weeks
• Phase 5: Update Documentation - 1 week
• Phase 6: Update Dependencies - 1-2 days
• Phase 7: Update CI/CD - 2-3 days

Total Estimated Time: 4-6 weeks

Success Criteria

• All tests pass with new implementation
• RPC service successfully sends transactions to propagation service
• External distribution library is functional and documented
• Documentation is complete and accurate
• Migration guide helps clients update smoothly
• No performance regression for simple use cases
• CI/CD pipeline works with new implementation

Related Files

Files to Modify

• services/rpc/handlers.go - Update sendrawtransaction handler
• services/rpc/Server.go - Add propagation client, remove distributor references
• settings/settings.go - Remove distributor settings
• settings/interface.go - Remove distributor settings from interface
• settings.conf - Remove distributor configuration

Files to Remove

• services/rpc/Distributor.go
• services/rpc/Distributor_test.go
• services/rpc/Distributor_comprehensive_test.go

Files to Create

• External library repository/package
• docs/refactoring/remove-rpc-distributor.md (this file)
• docs/migration/rpc-distributor-removal.md

Test Files to Update

• services/rpc/handlers_test.go
• services/rpc/handlers_additional_test.go
• services/rpc/server_test.go
• test/utils/transaction_helper.go
• test/utils/testenv.go
• test/utils/helper.go
• Various integration and E2E test files

Documentation Files to Update

• docs/topics/services/rpc.md
• docs/howto/submitting_transactions.md
• docs/topics/services/img/plantuml/rpc/*.puml
• README.md (if references RPC features)

References

• Current Distributor implementation: services/rpc/Distributor.go
• Propagation client: services/propagation/Client.go
• RPC handler: services/rpc/handlers.go:710-756
• Settings: settings/settings.go:406-410