Skip to content

Add IdleTimeout to default docker.m Aerospike utxostore#1049

Merged
oskarszoon merged 2 commits into
bsv-blockchain:mainfrom
oskarszoon:fix/aerospike-idle-conns
Jun 8, 2026
Merged

Add IdleTimeout to default docker.m Aerospike utxostore#1049
oskarszoon merged 2 commits into
bsv-blockchain:mainfrom
oskarszoon:fix/aerospike-idle-conns

Conversation

@oskarszoon

@oskarszoon oskarszoon commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

What

Add IdleTimeout=60s to the utxostore.docker.m Aerospike connection URL default.

Why

The Aerospike Go client's ClientPolicy.IdleTimeout defaults to 0 — a live client never reaps its own idle sockets. The docker.m UTXO store URL never set it, so idle connections are never cleaned up on the client side.

Combined with a server that isn't reaping either (proto-fd-idle-ms is off by default and deprecated in the 8.x line), connections only ever accumulate toward proto-fd-max and never come back down. On a downed cluster — all Teranode containers stopped, zero live clients — the Aerospike server was observed holding ~21k orphan client connections (client_connections_opened − client_connections_closed) against a proto-fd-max of 30k. Booting Teranode on top of that pool walks straight into the ceiling and connections start failing.

IdleTimeout=60s makes a live client reap its own excess idle connections after 60s.

Behavior notes

  • The MinConnectionsPerNode=16 reserve is exempt from idle reaping — verified in the client pool (connection_heap.go DropIdle only drops connections above the min floor; reserve conns get their idle deadline refreshed each tend). The reserve stays warm; only connections above 16 are reaped.
  • Client IdleTimeout must stay below the server's proto-fd-idle-ms so the client reaps its own socket before the server does — grabbing a socket the server already reaped throws. Pair this with a server-side proto-fd-idle-ms higher than 60s (our deployment runs 120000 / 2 min).
  • The IBD hot path is unaffected: under load connections are continuously in use and never sit idle long enough to reap. This only governs quiet-period pool shrink and dead-socket cleanup.

Scope

docker.m profile only. The other Aerospike UTXO URLs (dev.legacy, teratestnet) also omit IdleTimeout but run single-node localhost dev setups where orphan accumulation hasn't bitten; left unchanged.

@github-actions

github-actions Bot commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🤖 Claude Code Review

Status: Complete

Current Review:
No critical issues found. The change is minimal, correct, and well-justified.

Minor documentation gap:
The IdleTimeout parameter is now used in production config (utxostore.docker.m) but is not documented in docs/references/settings/stores/aerospike_settings.md table (lines 130-140). Consider adding it alongside the other UTXO Store URL parameters for completeness.

Change verified:

  • Parameter correctly parsed by util/aerospike.go:333 (getQueryDuration)
  • Applied to ClientPolicy.IdleTimeout via line 333
  • Scope limited to docker.m profile only as intended
  • Technical rationale sound (client reaps before server, reserves exempt from reaping)

@github-actions

github-actions Bot commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Comparison Report

Baseline: main (unknown)

Current: PR-1049 (544e902)

Summary

  • Regressions: 0
  • Improvements: 0
  • Unchanged: 132
  • Significance level: p < 0.05
All benchmark results (sec/op)
Benchmark Baseline Current Change p-value
_NewBlockFromBytes-4 1.772µ 1.767µ ~ 0.700
SplitSyncedParentMap_SetIfNotExists/256_buckets-4 61.72n 61.85n ~ 1.000
SplitSyncedParentMap_SetIfNotExists/16_buckets-4 61.78n 61.77n ~ 1.000
SplitSyncedParentMap_SetIfNotExists/1_bucket-4 61.78n 61.71n ~ 1.000
SplitSyncedParentMap_ConcurrentSetIfNotExists/256_buckets... 31.21n 30.53n ~ 0.400
SplitSyncedParentMap_ConcurrentSetIfNotExists/16_buckets_... 54.29n 53.74n ~ 0.400
SplitSyncedParentMap_ConcurrentSetIfNotExists/1_bucket_pa... 126.5n 116.9n ~ 0.200
MiningCandidate_Stringify_Short-4 259.8n 261.4n ~ 0.400
MiningCandidate_Stringify_Long-4 1.902µ 1.918µ ~ 0.100
MiningSolution_Stringify-4 992.6n 1002.0n ~ 0.100
BlockInfo_MarshalJSON-4 1.814µ 1.828µ ~ 0.100
NewFromBytes-4 130.4n 130.4n ~ 0.800
AddTxBatchColumnar_Validation-4 2.488µ 2.445µ ~ 0.200
OffsetValidationLoop-4 642.3n 641.3n ~ 0.800
Mine_EasyDifficulty-4 60.36µ 60.54µ ~ 0.700
Mine_WithAddress-4 6.772µ 7.448µ ~ 0.100
BlockAssembler_AddTx-4 0.02203n 0.02338n ~ 1.000
AddNode-4 9.944 10.433 ~ 0.200
AddNodeWithMap-4 10.36 10.32 ~ 0.400
DirectSubtreeAdd/4_per_subtree-4 57.85n 57.67n ~ 1.000
DirectSubtreeAdd/64_per_subtree-4 29.92n 30.19n ~ 0.400
DirectSubtreeAdd/256_per_subtree-4 29.21n 29.07n ~ 0.700
DirectSubtreeAdd/1024_per_subtree-4 27.84n 27.93n ~ 0.400
DirectSubtreeAdd/2048_per_subtree-4 27.47n 27.52n ~ 0.400
SubtreeProcessorAdd/4_per_subtree-4 280.2n 316.4n ~ 0.100
SubtreeProcessorAdd/64_per_subtree-4 272.7n 279.4n ~ 0.400
SubtreeProcessorAdd/256_per_subtree-4 267.2n 299.4n ~ 0.100
SubtreeProcessorAdd/1024_per_subtree-4 265.9n 280.5n ~ 0.100
SubtreeProcessorAdd/2048_per_subtree-4 266.7n 278.2n ~ 0.100
SubtreeProcessorRotate/4_per_subtree-4 271.1n 275.1n ~ 0.200
SubtreeProcessorRotate/64_per_subtree-4 266.7n 271.7n ~ 0.100
SubtreeProcessorRotate/256_per_subtree-4 270.5n 274.3n ~ 0.400
SubtreeProcessorRotate/1024_per_subtree-4 268.2n 273.0n ~ 0.100
SubtreeNodeAddOnly/4_per_subtree-4 53.47n 53.94n ~ 0.400
SubtreeNodeAddOnly/64_per_subtree-4 34.07n 34.06n ~ 0.500
SubtreeNodeAddOnly/256_per_subtree-4 33.01n 33.05n ~ 0.700
SubtreeNodeAddOnly/1024_per_subtree-4 32.61n 32.48n ~ 0.700
SubtreeCreationOnly/4_per_subtree-4 99.30n 120.90n ~ 0.100
SubtreeCreationOnly/64_per_subtree-4 458.8n 506.2n ~ 0.700
SubtreeCreationOnly/256_per_subtree-4 1.395µ 1.354µ ~ 0.100
SubtreeCreationOnly/1024_per_subtree-4 4.502µ 5.117µ ~ 0.100
SubtreeCreationOnly/2048_per_subtree-4 8.383µ 9.030µ ~ 0.100
SubtreeProcessorOverheadBreakdown/64_per_subtree-4 268.6n 270.6n ~ 0.200
SubtreeProcessorOverheadBreakdown/1024_per_subtree-4 267.1n 276.3n ~ 0.100
ParallelGetAndSetIfNotExists/1k_nodes-4 11.01m 13.70m ~ 0.100
ParallelGetAndSetIfNotExists/10k_nodes-4 14.03m 17.34m ~ 0.100
ParallelGetAndSetIfNotExists/50k_nodes-4 16.37m 20.83m ~ 0.100
ParallelGetAndSetIfNotExists/100k_nodes-4 19.03m 17.16m ~ 0.100
SequentialGetAndSetIfNotExists/1k_nodes-4 10.63m 14.64m ~ 0.100
SequentialGetAndSetIfNotExists/10k_nodes-4 14.11m 19.04m ~ 0.100
SequentialGetAndSetIfNotExists/50k_nodes-4 18.37m 19.72m ~ 0.400
SequentialGetAndSetIfNotExists/100k_nodes-4 25.69m 25.70m ~ 1.000
ProcessOwnBlockSubtreeNodesParallel/1k_nodes-4 12.67m 13.93m ~ 0.100
ProcessOwnBlockSubtreeNodesParallel/10k_nodes-4 13.84m 14.40m ~ 0.100
ProcessOwnBlockSubtreeNodesParallel/100k_nodes-4 18.66m 18.89m ~ 0.400
ProcessOwnBlockSubtreeNodesSequential/1k_nodes-4 14.28m 13.12m ~ 1.000
ProcessOwnBlockSubtreeNodesSequential/10k_nodes-4 14.22m 14.88m ~ 0.100
ProcessOwnBlockSubtreeNodesSequential/100k_nodes-4 49.15m 50.61m ~ 0.100
DiskTxMap_SetIfNotExists-4 3.429µ 3.445µ ~ 1.000
DiskTxMap_SetIfNotExists_Parallel-4 3.291µ 3.234µ ~ 0.700
DiskTxMap_ExistenceOnly-4 308.3n 304.7n ~ 0.100
Queue-4 187.2n 189.7n ~ 0.100
AtomicPointer-4 4.438n 4.753n ~ 0.200
TxMapSetIfNotExists-4 52.13n 52.38n ~ 0.100
TxMapSetIfNotExistsDuplicate-4 39.75n 39.82n ~ 1.000
ChannelSendReceive-4 568.8n 566.0n ~ 0.200
CalcBlockWork-4 513.7n 512.9n ~ 1.000
CalculateWork-4 704.0n 695.2n ~ 0.200
CheckOldBlockIDs/on-chain-prefetch/1000-4 68.75µ 58.68µ ~ 0.200
CheckOldBlockIDs/off-chain-prefetch/1000-4 49.81µ 49.51µ ~ 1.000
CheckOldBlockIDs/on-chain-prefetch/10000-4 416.0µ 449.5µ ~ 0.100
CheckOldBlockIDs/off-chain-prefetch/10000-4 346.4µ 352.0µ ~ 0.100
BuildBlockLocatorString_Helpers/Size_10-4 1.348µ 1.340µ ~ 0.300
BuildBlockLocatorString_Helpers/Size_100-4 12.87µ 12.79µ ~ 0.100
BuildBlockLocatorString_Helpers/Size_1000-4 126.4µ 126.4µ ~ 1.000
CatchupWithHeaderCache-4 104.4m 104.1m ~ 0.100
_prepareTxsPerLevel-4 408.1m 404.3m ~ 0.400
_prepareTxsPerLevelOrdered-4 3.473m 3.659m ~ 0.200
_prepareTxsPerLevel_Comparison/Original-4 404.9m 409.0m ~ 0.200
_prepareTxsPerLevel_Comparison/Optimized-4 3.686m 3.608m ~ 0.700
_BufferPoolAllocation/16KB-4 3.856µ 4.456µ ~ 0.200
_BufferPoolAllocation/32KB-4 11.56µ 10.06µ ~ 0.700
_BufferPoolAllocation/64KB-4 16.47µ 15.99µ ~ 0.400
_BufferPoolAllocation/128KB-4 31.96µ 32.07µ ~ 1.000
_BufferPoolAllocation/512KB-4 122.9µ 112.2µ ~ 0.400
_BufferPoolConcurrent/32KB-4 20.04µ 18.82µ ~ 0.100
_BufferPoolConcurrent/64KB-4 30.49µ 31.20µ ~ 0.100
_BufferPoolConcurrent/512KB-4 146.4µ 148.3µ ~ 0.700
_SubtreeDeserializationWithBufferSizes/16KB-4 673.4µ 650.6µ ~ 0.700
_SubtreeDeserializationWithBufferSizes/32KB-4 670.8µ 676.7µ ~ 0.700
_SubtreeDeserializationWithBufferSizes/64KB-4 661.5µ 652.4µ ~ 0.700
_SubtreeDeserializationWithBufferSizes/128KB-4 670.4µ 708.9µ ~ 0.700
_SubtreeDeserializationWithBufferSizes/512KB-4 684.0µ 608.8µ ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/16KB-4 36.81m 37.44m ~ 0.200
_SubtreeDataDeserializationWithBufferSizes/32KB-4 36.70m 37.98m ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/64KB-4 36.49m 37.79m ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/128KB-4 36.47m 37.25m ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/512KB-4 36.30m 37.53m ~ 0.100
_PooledVsNonPooled/Pooled-4 833.6n 831.5n ~ 0.200
_PooledVsNonPooled/NonPooled-4 7.711µ 8.064µ ~ 0.400
_MemoryFootprint/Current_512KB_32concurrent-4 6.823µ 7.584µ ~ 0.100
_MemoryFootprint/Proposed_32KB_32concurrent-4 9.942µ 9.777µ ~ 0.700
_MemoryFootprint/Alternative_64KB_32concurrent-4 9.424µ 9.403µ ~ 1.000
SubtreeSizes/10k_tx_4_per_subtree-4 1.264m 1.265m ~ 1.000
SubtreeSizes/10k_tx_16_per_subtree-4 293.5µ 297.8µ ~ 0.700
SubtreeSizes/10k_tx_64_per_subtree-4 70.88µ 70.46µ ~ 0.400
SubtreeSizes/10k_tx_256_per_subtree-4 17.44µ 17.56µ ~ 0.700
SubtreeSizes/10k_tx_512_per_subtree-4 8.744µ 8.702µ ~ 0.400
SubtreeSizes/10k_tx_1024_per_subtree-4 4.371µ 4.336µ ~ 1.000
SubtreeSizes/10k_tx_2k_per_subtree-4 2.136µ 2.140µ ~ 0.200
BlockSizeScaling/10k_tx_64_per_subtree-4 68.91µ 69.28µ ~ 1.000
BlockSizeScaling/10k_tx_256_per_subtree-4 17.32µ 17.36µ ~ 1.000
BlockSizeScaling/10k_tx_1024_per_subtree-4 4.310µ 4.314µ ~ 1.000
BlockSizeScaling/50k_tx_64_per_subtree-4 365.9µ 368.8µ ~ 0.700
BlockSizeScaling/50k_tx_256_per_subtree-4 86.78µ 87.43µ ~ 0.200
BlockSizeScaling/50k_tx_1024_per_subtree-4 21.12µ 21.54µ ~ 0.700
SubtreeAllocations/small_subtrees_exists_check-4 149.0µ 148.7µ ~ 0.400
SubtreeAllocations/small_subtrees_data_fetch-4 156.1µ 158.3µ ~ 0.100
SubtreeAllocations/small_subtrees_full_validation-4 302.9µ 309.5µ ~ 0.100
SubtreeAllocations/medium_subtrees_exists_check-4 9.032µ 8.831µ ~ 0.200
SubtreeAllocations/medium_subtrees_data_fetch-4 9.258µ 9.243µ ~ 0.400
SubtreeAllocations/medium_subtrees_full_validation-4 17.28µ 17.31µ ~ 1.000
SubtreeAllocations/large_subtrees_exists_check-4 2.074µ 2.087µ ~ 0.300
SubtreeAllocations/large_subtrees_data_fetch-4 2.204µ 2.191µ ~ 0.200
SubtreeAllocations/large_subtrees_full_validation-4 4.275µ 4.318µ ~ 0.200
StoreBlock_Sequential/BelowCSVHeight-4 334.8µ 332.3µ ~ 0.100
StoreBlock_Sequential/AboveCSVHeight-4 337.4µ 341.2µ ~ 0.700
GetUtxoHashes-4 267.3n 264.9n ~ 0.400
GetUtxoHashes_ManyOutputs-4 44.40µ 43.42µ ~ 0.100
_NewMetaDataFromBytes-4 228.3n 227.6n ~ 0.700
_Bytes-4 400.3n 402.1n ~ 0.700
_MetaBytes-4 138.5n 138.5n ~ 0.800

Threshold: >10% with p < 0.05 | Generated: 2026-06-08 09:16 UTC

@sonarqubecloud

sonarqubecloud Bot commented Jun 8, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

ordishs
ordishs approved these changes Jun 8, 2026
View reviewed changes

@ordishs ordishs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Verified the change is wired correctly:

  • IdleTimeout is parsed (util/aerospike.go:333) and mapped onto policy.IdleTimeout — not silently ignored.
  • getQueryDuration uses time.ParseDuration, so 60s parses to 60 seconds. Note an integer (60) would be rejected here, so the duration syntax is the right form.
  • The param sits in the top-level Aerospike URL query, not inside the percent-encoded externalStore sub-URL — no nesting hazard, and query-param order is irrelevant.

One durable risk worth flagging: correctness depends on the server's proto-fd-idle-ms staying > 60s (deployment runs 120s). That invariant lives only in the PR description — it isn't enforced anywhere in this repo. If a future deployment lowers the server value below 60s, clients could grab server-reaped sockets and throw.

Optional (non-blocking): consider an inline comment in settings.conf capturing the dependency, e.g.

# IdleTimeout must stay below server proto-fd-idle-ms (deployment: 120s)

Otherwise solid and ready to merge.

@oskarszoon oskarszoon merged commit bf19c99 into bsv-blockchain:main Jun 8, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ordishs ordishs ordishs approved these changes

@sugh01 sugh01 sugh01 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@oskarszoon @ordishs @sugh01