Summary

Raise ConnectionQueueSize from 16 to 128 (and MinConnectionsPerNode from 8 to 16) on the utxostore.docker.m URL. docker.m (docker-microservice) context only — used by teranode-quickstart and other docker-based deployments. Other contexts unchanged.

Why

ConnectionQueueSize=16 + LimitConnectionsToQueueSize=true is too tight for the default pruner partition-worker fanout, and for legacy block-processing batch ops under mainnet IBD load.

The pruner detects this and emits:

WARN | pruner/pruner_service.go:425 | utxos | 
Pruner concurrency would exhaust Aerospike connection pool. 
Max pruner connections: 64, ConnectionQueueSize: 16, Recommended max: 11. 
Auto-adjusting pruner_utxoChunkGroupLimit from 1 to 1 to prevent exhaustion.

It auto-throttles pruner_utxoChunkGroupLimit but not pruner_utxoPartitionQueries (the outer 32-worker fanout). Even at chunk_group_limit=1, 32 partition workers × 1 = 32 concurrent ops, which still exceeds the 16-conn pool. Every pruner batch then errors with NO_AVAILABLE_CONNECTIONS_TO_NODE followed by TIMEOUT.

Observed

bsva-ovh-teranode-eu-3, mainnet, v0.15.2-beta-1, height ~795,360.

Cascade with the old pool size:

1. Legacy retries on fat blocks (compounded by legacy: createUtxos calls SetMinedMulti with unbounded slice — stalls aerospike on fat blocks (regression from #854) #936 before its fix) leak some client connections faster than they close
2. Pool saturates at 16
3. Pruner can't issue parent-update writes — perpetually NO_AVAILABLE_CONNECTIONS_TO_NODE
4. Spent UTXOs aren't deleted; objects climb to 832M
5. stop-writes-used-pct=70 (inherited from evict-used-pct=70) trips
6. Aerospike enters stop_writes=true + hwm_breached=true
7. Pruner needs to write to delete things, but stop_writes blocks writes → deadlock

The 16-conn ceiling was the bottleneck, not the aerospike server: proto-fd-max defaults to 15,000, and the actual server-side connection count peaked around 72 (across all teranode services combined) before the lockup, so there's plenty of server headroom.

Why 128 specifically

With the default pruner_utxoPartitionQueries=0 (auto-detect, ~32 workers on a typical host) and a chunk_group_limit that can rise to its 10 default once the pool is comfortable, peak pruner concurrency is ~320 ops. 128 connections gives that 2.5× headroom for the auto-adjust to land at 5–8 chunk groups rather than being pinned at 1. Other utxostore consumers (legacy, blockvalidation, subtreevalidation, blockassembly, blockchain, propagation, asset, pruner) each get their own pool; at 128/pool × 8 clients = 1,024 total potential connections, well under proto-fd-max=15000.

MinConnectionsPerNode raised proportionally (8 → 16) so warm-up provisions a reasonable baseline.

Scope

Only utxostore.docker.m changes. The commented-out utxostore.docker template right below remains at ConnectionQueueSize=32 (it's a template, not active). Operator/k8s contexts unchanged.

Verification

• Config diff is one line, ConnectionQueueSize=16 → 128, MinConnectionsPerNode=8 → 16
• Operator validation: bring up a docker.m deployment (e.g. teranode-quickstart), confirm docker exec aerospike asinfo -v statistics | grep client_connections reflects the larger ceiling and the pruner WARN at pruner_service.go:425 no longer auto-throttles to chunk_group_limit=1

Related

• legacy: createUtxos calls SetMinedMulti with unbounded slice — stalls aerospike on fat blocks (regression from #854) #936 — createUtxos unbounded SetMinedMulti (one of the connection-burner paths upstream of this issue)
• blockvalidation: OOM during ttn catch-up sync — 70% of heap in go-bt tx/output decode #920 — go-bt arena fix (already merged in v0.15.2)

Not in this PR

• The underlying aerospike-client-go/v8 connection-handling on timeout (InDoubt: true paths) leaks slower than steady-state churn returns them. Worth a separate investigation. Bigger pool buys time, doesn't eliminate the leak.
• evict-used-pct=70 in config/aerospike.conf template silently anchors stop-writes-used-pct=70 even though eviction is a no-op for default-ttl 0. Separate concern; affects quickstart configs.
• The pruner auto-adjust should also throttle pruner_utxoPartitionQueries, not just pruner_utxoChunkGroupLimit. Separate fix.