Summary

Hotfix companion to #828 / #830: serialize SetTxMined operations on the block-validator side so they don't pile up and cause Aerospike-client mutex contention to collapse throughput to zero.

Production evidence (dev-scale-1-scale-1 on commit fdaeadb7e)

Block-validator pod was reported as "stopped doing setTxMined". It had not actually stopped — it had stacked up:

Live pod state at the time:

• 5.68 CPU cores, 107 GB live heap
• pprof: many goroutines blocked in aerospike-client-go/v8/internal/atomic/map.Map.Set on sync.RWMutex.Lock() — the per-node nodeStats map mutex
• All 6 Aerospike pods healthy, no errors in client logs

Root cause

processBlockMinedNotSet (in BlockValidation.go) spawns one goroutine per block via errgroup.Go for every block the blockchain reports as still needing mined_set. It runs:

• once at startup (recovery)
• every minute via the periodic ticker

Each SetTxMined operation fans out into per-subtree workers and 1024-key Aerospike batches. The Aerospike Go client serializes every batch result through a single per-node RWMutex.Lock() updating its nodeStats histogram. With multiple parallel SetTxMined operations layered on top, you get a thundering herd on that one mutex and effective throughput collapses.

Meanwhile a separate setMinedChan worker (BlockValidation.go:501) already exists and processes blocks serially — including the MinedSet guard, tryClaimBlockForSetMined dedup, retry-on-error, and cleanup. The polling path in processBlockMinedNotSet was duplicating all of that logic in parallel goroutines, defeating the worker's serialization.

Change

processBlockMinedNotSet now enqueues block hashes onto setMinedChan instead of spawning goroutines. The worker handles everything else, naturally serializing operations.

- for _, block := range blocksMinedNotSet {
-     blockHash := block.Hash()
-     if !u.tryClaimBlockForSetMined(blockHash) { continue }
-     g.Go(func() error {
-         defer func() { u.blockHashesCurrentlyValidated.Delete(*blockHash) }()
-         // ... fetch header, call setTxMinedStatus, retry on err ...
-     })
- }
+ for _, block := range blocksMinedNotSet {
+     blockHash := block.Hash()
+     select {
+     case <-ctx.Done():
+         return
+     case u.setMinedChan <- blockHash:
+     }
+ }

Net: 1 file, -50 / +34 lines (more deletions than additions because the duplicated worker logic collapses).

Behavior changes

• At most one SetTxMined runs at a time per pod. This is the fix.
• Startup recovery is now asynchronous — start() returns before queued blocks have completed. No callers depend on synchronous startup completion (verified — NewBlockValidation runs start() in a fire-and-forget goroutine).
• The errgroup.Group parameter remains in the signature for caller-symmetry with processSubtreesNotSet (called from the same ticker), but it is unused — marked _ to make that explicit.
• setMinedChan capacity is 1000; a backlog beyond that would block the ticker. The existing worker-error retry path already has the same characteristic, so this isn't a new failure mode.

Why not just disable expressions / cut over the expressions path?

This fix is complementary to those, not a replacement. From the production analysis on PR #828:

1. Disable aerospike_enable_setmined_filter_expressions — moves to the PR aerospike: optional native operate-path for mod-teranode UDFs #821 native-op-cutover'd path. Might reduce per-batch latency, but doesn't eliminate the parallel-thundering-herd.
2. Cut over SetMinedMultiWithExpressions to native ops — bigger change, future PR.
3. (Upstream) patch aerospike-client-go/v8's nodeStats to be lock-free — the real fix, separate effort.

This PR addresses the amplifier (parallelism). Even if the per-op cost stays the same, serializing means:

• Only 1 SetTxMined competes for the Aerospike mutex at a time.
• Memory stays bounded (no piled-up batch records — we observed 107 GB heap from accumulated parallel state).
• Operations complete in deterministic order — which matters because they retry on failure by re-queuing into setMinedChan.

Test plan

• go build ./... — clean
• go vet ./services/blockvalidation/... — clean
• Pre-commit hooks (gofmt, gci, golangci-lint) — pass
• Existing TestBlockValidation* tests pass except one pre-existing timeout (TestBlockValidation_InvalidParentBlock, also fails on unmodified branch — unrelated)
• Deploy to dev-scale-1 and confirm:
  • Block-validator memory stays bounded (no 107 GB heaps)
  • [setTxMined][...] setting tx mined log lines pair with corresponding completion
  • Pending blocks_mined_not_set count drains over time

Risks

• Async startup: if some operator or test depended on BlockValidation.start() not returning until startup recovery completed, that assumption breaks. Mitigation: I checked the only caller (NewBlockValidation) and it runs start() in a fire-and-forget goroutine.
• Channel buffer overflow: at startup with >1000 pending blocks, the loop blocks until the worker drains. Worker starts shortly after g.Wait() returns. Acceptable for any realistic backlog.
• Rollback: pure revert of one commit.

Targeting

Base: feat/teranode-native-ops (where the deployed pods are running). Will need to also apply to main once #828 merges.