Skip to content

fix(legacy): merge blockID into pre-existing tx in createUtxos#854

Merged
oskarszoon merged 1 commit into
bsv-blockchain:mainfrom
oskarszoon:fix/legacy-merge-blockid-on-tx-exists
May 13, 2026
Merged

fix(legacy): merge blockID into pre-existing tx in createUtxos#854
oskarszoon merged 1 commit into
bsv-blockchain:mainfrom
oskarszoon:fix/legacy-merge-blockid-on-tx-exists

Conversation

@oskarszoon

Copy link
Copy Markdown
Contributor

Summary

Fixes BLOCK_INVALID (11): parent transaction X of tx Y has no block IDs that stalled both bsva-ovh-teranode-eu-3 (mainnet, height 302,681) and bsva-ovh-teranode-ttn-eu-4 (testnet, height 1,512,911) within ~8 minutes of each other on 2026-05-12 after upgrade to v0.15.0-beta-10.

Root cause

PR #711 (commit 95dbe32) added a quickValidation fast path: legacy netsync pre-assigns a block ID, bakes it into UTXO records during createUtxos, and skips the async setTxMinedStatus step by calling AddBlock(WithID, WithMinedSet(true)).

The fast path relies on createUtxos putting the block ID into every tx's BlockIDs at creation time. But when utxoStore.Create returned ErrTxExists for a tx that pre-existed (propagation arrival, prior crashed attempt, or the pre-fast-path subtreeValidation route), the previous code logged at Debug and returned nil:

if errors.Is(err, errors.ErrTxExists) {
    sm.logger.Debugf("failed to create utxo for tx %s: %s", txHash.String(), err)
} else {
    return err
}

The new block's ID was never merged into the existing tx's BlockIDs. Subsequent blocks failed validOrderAndBlessed in model/Block.go:1070-1072 (getParentTxMetaBlockIDs) and cascaded — every descendant block marked invalid, and the setTxMined recovery path with unsetMined=true then cleared BlockIDs on txs in those invalid blocks, deepening the corruption.

Pre-v0.15 the async setTxMinedStatusSetMinedMulti path performed this merge implicitly after block accept. The fast path short-circuited that.

Fix

Mirror the pattern from services/blockvalidation/quick_validate.go:1106-1161 createAndSpendUTXOsForBatch:

  1. During the parallel Create pass, collect tx hashes that returned ErrTxExists.
  2. After the errgroup waits, issue one SetMinedMulti to merge the new blockID into all pre-existing records.

Reproduction timeline (eu-3, from Coralogix archive)

  • 14:56:19Z legacy on pre-upgrade version starts processing block 302680 via the slow path (extendTransactionssubtreeValidation); subtreeValidation creates txs in the utxo store with empty BlockIDs.
  • 14:56:24Z legacy errors: SUBTREE_ERROR (29): failed to check subtree → rpc Canceled (user-initiated upgrade restart).
  • 14:56:45Z v0.15.0-beta-10 starts.
  • 14:56:55Z legacy fast path createUtxos runs for block 302680. Tx 54ea96a0… already exists in store with empty BlockIDs → ErrTxExists → silently dropped.
  • 14:56:56Z blockvalidation AddBlock stores 302680 with mined_set=true; setMinedChan worker logs "block already has mined_set true, skipping setTxMined".
  • 14:56:58Z blockvalidation processes 302681; child tx 543094cd… has parent 54ea96a0… (which lives in 302680's subtree); getParentTxMetaBlockIDs finds it in store with empty BlockIDs → BLOCK_INVALID. Cascade follows.

Test plan

  • go test -race ./services/legacy/netsync/... — 50 passed, including new TestSyncManager_createUtxos_MergesBlockIDsForExistingTxs which pre-creates a tx in a sqlitememory utxo store with empty BlockIDs, runs createUtxos, and asserts the new blockID was merged in.
  • go vet ./services/legacy/netsync/... ./model/...
  • golangci-lint run ./services/legacy/netsync/...
  • staticcheck ./services/legacy/netsync/...
  • Verify recovery on eu-3 / ttn-eu-4: deploy patched binary, then reconsiderblock on the invalid chain tip. Validation will re-execute, this time merging block IDs into the orphaned txs.

Notes

  • Both affected hosts were resetting per user direction; the fix prevents recurrence.
  • Other callers of utxoStore.Create with WithMinedBlockInfo should be audited for the same ErrTxExists silent-drop pattern; the equivalent path in services/blockvalidation/quick_validate.go already handles it correctly via a Phase 1.5 batch update.

The legacy quickValidation fast path stores blocks with mined_set=true
upfront and skips the async setTxMinedStatus step, relying on
createUtxos to bake BlockIDs into each tx at creation time. When
utxoStore.Create returned ErrTxExists for a tx that pre-existed
(propagation arrival, prior crashed attempt, or the slow-path
subtreeValidation route), the previous code logged at Debug and
returned nil — the new block's ID was never merged into the existing
tx's BlockIDs.

Subsequent blocks then failed validation in model/Block.go
getParentTxMetaBlockIDs with "parent transaction X of tx Y has no
block IDs" (BLOCK_INVALID), cascading to mark every descendant block
invalid. Observed on mainnet eu-3 (height 302681) and testnet
ttn-eu-4 (height 1512911) after upgrade to v0.15.0-beta-10.

Mirror the pattern from services/blockvalidation/quick_validate.go
createAndSpendUTXOsForBatch: collect pre-existing tx hashes during
the parallel Create pass, then issue one SetMinedMulti to merge the
blockID into all of them. v0.14.x performed this merge implicitly
via the async setTxMined → SetMinedMulti path; the fast path
short-circuited that and lost the merge.

Regression introduced by PR bsv-blockchain#711 (commit 95dbe32) which added the
pre-assigned blockID + mined_set=true fast path.
@github-actions

github-actions Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

🤖 Claude Code Review

Status: Complete


Current Review:

The fix correctly addresses the root cause where pre-existing transactions were not getting the new blockID merged into their BlockIDs field. The implementation mirrors the pattern from quick_validate.go and includes comprehensive test coverage.

One minor observation: The code at handle_block.go:671 appends &txHash where txHash is a goroutine-captured value. While this should be safe due to Go escape analysis (the variable will be heap-allocated), it differs slightly from the reference implementation which calls tx.TxIDChainHash() to get a fresh pointer. Both approaches should work correctly.

No blocking issues found. The fix follows project patterns and includes proper error handling, logging, and test coverage.

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Comparison Report

Baseline: main (unknown)

Current: PR-854 (6e2cb87)

Summary

  • Regressions: 0
  • Improvements: 0
  • Unchanged: 142
  • Significance level: p < 0.05
All benchmark results (sec/op)
Benchmark Baseline Current Change p-value
_NewBlockFromBytes-4 1.738µ 2.010µ ~ 0.400
SplitSyncedParentMap_SetIfNotExists/256_buckets-4 59.58n 59.24n ~ 1.000
SplitSyncedParentMap_SetIfNotExists/16_buckets-4 59.30n 59.37n ~ 0.700
SplitSyncedParentMap_SetIfNotExists/1_bucket-4 59.17n 59.24n ~ 0.700
SplitSyncedParentMap_ConcurrentSetIfNotExists/256_buckets... 34.91n 34.68n ~ 0.700
SplitSyncedParentMap_ConcurrentSetIfNotExists/16_buckets_... 59.28n 59.38n ~ 0.500
SplitSyncedParentMap_ConcurrentSetIfNotExists/1_bucket_pa... 150.5n 154.2n ~ 0.100
MiningCandidate_Stringify_Short-4 259.8n 259.0n ~ 0.700
MiningCandidate_Stringify_Long-4 1.782µ 1.785µ ~ 0.700
MiningSolution_Stringify-4 919.5n 921.2n ~ 0.400
BlockInfo_MarshalJSON-4 1.771µ 1.776µ ~ 0.400
NewFromBytes-4 150.8n 130.7n ~ 0.200
Mine_EasyDifficulty-4 67.35µ 68.63µ ~ 0.200
Mine_WithAddress-4 7.704µ 7.145µ ~ 0.100
DirectSubtreeAdd/4_per_subtree-4 59.00n 60.33n ~ 0.700
DirectSubtreeAdd/64_per_subtree-4 28.90n 28.81n ~ 0.700
DirectSubtreeAdd/256_per_subtree-4 27.43n 27.60n ~ 0.200
DirectSubtreeAdd/1024_per_subtree-4 26.21n 26.24n ~ 1.000
DirectSubtreeAdd/2048_per_subtree-4 25.91n 25.99n ~ 0.300
SubtreeProcessorAdd/4_per_subtree-4 286.4n 281.5n ~ 0.400
SubtreeProcessorAdd/64_per_subtree-4 277.9n 276.4n ~ 0.700
SubtreeProcessorAdd/256_per_subtree-4 277.1n 271.4n ~ 0.100
SubtreeProcessorAdd/1024_per_subtree-4 272.2n 266.4n ~ 0.100
SubtreeProcessorAdd/2048_per_subtree-4 270.8n 267.2n ~ 0.100
SubtreeProcessorRotate/4_per_subtree-4 273.9n 273.3n ~ 0.400
SubtreeProcessorRotate/64_per_subtree-4 272.2n 272.9n ~ 0.400
SubtreeProcessorRotate/256_per_subtree-4 272.1n 272.2n ~ 1.000
SubtreeProcessorRotate/1024_per_subtree-4 271.5n 271.7n ~ 0.800
SubtreeNodeAddOnly/4_per_subtree-4 54.08n 54.65n ~ 0.100
SubtreeNodeAddOnly/64_per_subtree-4 34.34n 34.47n ~ 0.100
SubtreeNodeAddOnly/256_per_subtree-4 33.32n 33.52n ~ 0.100
SubtreeNodeAddOnly/1024_per_subtree-4 32.75n 32.63n ~ 0.100
SubtreeCreationOnly/4_per_subtree-4 113.3n 112.9n ~ 1.000
SubtreeCreationOnly/64_per_subtree-4 401.1n 394.7n ~ 0.200
SubtreeCreationOnly/256_per_subtree-4 1.337µ 1.436µ ~ 0.100
SubtreeCreationOnly/1024_per_subtree-4 4.398µ 4.526µ ~ 0.100
SubtreeCreationOnly/2048_per_subtree-4 8.086µ 8.048µ ~ 0.400
SubtreeProcessorOverheadBreakdown/64_per_subtree-4 269.9n 271.2n ~ 0.100
SubtreeProcessorOverheadBreakdown/1024_per_subtree-4 272.2n 268.7n ~ 0.100
ParallelGetAndSetIfNotExists/1k_nodes-4 608.6µ 807.8µ ~ 0.100
ParallelGetAndSetIfNotExists/10k_nodes-4 1.357m 1.555m ~ 0.700
ParallelGetAndSetIfNotExists/50k_nodes-4 6.678m 6.605m ~ 0.700
ParallelGetAndSetIfNotExists/100k_nodes-4 13.29m 13.18m ~ 0.200
SequentialGetAndSetIfNotExists/1k_nodes-4 657.3µ 660.0µ ~ 0.700
SequentialGetAndSetIfNotExists/10k_nodes-4 2.790m 2.767m ~ 0.700
SequentialGetAndSetIfNotExists/50k_nodes-4 10.48m 10.29m ~ 0.200
SequentialGetAndSetIfNotExists/100k_nodes-4 20.07m 20.02m ~ 0.700
ProcessOwnBlockSubtreeNodesParallel/1k_nodes-4 648.0µ 648.0µ ~ 1.000
ProcessOwnBlockSubtreeNodesParallel/10k_nodes-4 4.326m 4.255m ~ 0.100
ProcessOwnBlockSubtreeNodesParallel/100k_nodes-4 16.78m 16.77m ~ 0.100
ProcessOwnBlockSubtreeNodesSequential/1k_nodes-4 698.5µ 713.4µ ~ 0.100
ProcessOwnBlockSubtreeNodesSequential/10k_nodes-4 6.410m 6.066m ~ 0.100
ProcessOwnBlockSubtreeNodesSequential/100k_nodes-4 39.55m 38.10m ~ 0.100
DiskTxMap_SetIfNotExists-4 3.681µ 3.621µ ~ 0.100
DiskTxMap_SetIfNotExists_Parallel-4 3.473µ 3.351µ ~ 0.700
DiskTxMap_ExistenceOnly-4 316.4n 304.8n ~ 0.100
Queue-4 197.5n 197.1n ~ 0.700
AtomicPointer-4 4.427n 4.689n ~ 0.400
ReorgOptimizations/DedupFilterPipeline/Old/10K-4 851.8µ 898.9µ ~ 0.200
ReorgOptimizations/DedupFilterPipeline/New/10K-4 831.3µ 853.6µ ~ 0.100
ReorgOptimizations/AllMarkFalse/Old/10K-4 115.3µ 110.8µ ~ 0.700
ReorgOptimizations/AllMarkFalse/New/10K-4 61.98µ 62.30µ ~ 0.700
ReorgOptimizations/HashSlicePool/Old/10K-4 70.96µ 71.24µ ~ 1.000
ReorgOptimizations/HashSlicePool/New/10K-4 12.29µ 11.51µ ~ 0.100
ReorgOptimizations/NodeFlags/Old/10K-4 5.345µ 6.388µ ~ 0.200
ReorgOptimizations/NodeFlags/New/10K-4 1.854µ 2.107µ ~ 0.100
ReorgOptimizations/DedupFilterPipeline/Old/100K-4 9.791m 10.339m ~ 0.200
ReorgOptimizations/DedupFilterPipeline/New/100K-4 9.843m 10.093m ~ 0.100
ReorgOptimizations/AllMarkFalse/Old/100K-4 1.146m 1.159m ~ 0.700
ReorgOptimizations/AllMarkFalse/New/100K-4 685.9µ 687.4µ ~ 0.700
ReorgOptimizations/HashSlicePool/Old/100K-4 673.8µ 691.6µ ~ 0.200
ReorgOptimizations/HashSlicePool/New/100K-4 305.3µ 316.6µ ~ 1.000
ReorgOptimizations/NodeFlags/Old/100K-4 57.87µ 55.66µ ~ 0.100
ReorgOptimizations/NodeFlags/New/100K-4 20.64µ 19.85µ ~ 0.400
TxMapSetIfNotExists-4 51.72n 51.63n ~ 0.700
TxMapSetIfNotExistsDuplicate-4 38.08n 38.19n ~ 0.700
ChannelSendReceive-4 614.3n 606.1n ~ 0.300
BlockAssembler_AddTx-4 0.02912n 0.02819n ~ 1.000
AddNode-4 10.37 10.61 ~ 1.000
AddNodeWithMap-4 10.48 10.15 ~ 1.000
CalcBlockWork-4 490.0n 500.0n ~ 0.700
CalculateWork-4 665.7n 679.4n ~ 0.100
BuildBlockLocatorString_Helpers/Size_10-4 1.303µ 1.298µ ~ 0.400
BuildBlockLocatorString_Helpers/Size_100-4 12.67µ 12.33µ ~ 0.700
BuildBlockLocatorString_Helpers/Size_1000-4 127.0µ 123.4µ ~ 0.700
CatchupWithHeaderCache-4 104.6m 104.4m ~ 0.100
_BufferPoolAllocation/16KB-4 3.604µ 3.353µ ~ 0.200
_BufferPoolAllocation/32KB-4 7.238µ 6.803µ ~ 0.100
_BufferPoolAllocation/64KB-4 14.57µ 15.98µ ~ 0.700
_BufferPoolAllocation/128KB-4 29.11µ 28.92µ ~ 0.700
_BufferPoolAllocation/512KB-4 107.4µ 112.5µ ~ 0.400
_BufferPoolConcurrent/32KB-4 17.08µ 18.64µ ~ 0.100
_BufferPoolConcurrent/64KB-4 28.16µ 29.35µ ~ 0.100
_BufferPoolConcurrent/512KB-4 143.8µ 148.0µ ~ 0.100
_SubtreeDeserializationWithBufferSizes/16KB-4 636.0µ 674.1µ ~ 0.100
_SubtreeDeserializationWithBufferSizes/32KB-4 642.5µ 662.4µ ~ 0.200
_SubtreeDeserializationWithBufferSizes/64KB-4 648.4µ 681.1µ ~ 0.100
_SubtreeDeserializationWithBufferSizes/128KB-4 628.5µ 681.8µ ~ 0.100
_SubtreeDeserializationWithBufferSizes/512KB-4 653.0µ 682.7µ ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/16KB-4 35.49m 35.90m ~ 0.200
_SubtreeDataDeserializationWithBufferSizes/32KB-4 35.48m 35.83m ~ 0.100
_SubtreeDataDeserializationWithBufferSizes/64KB-4 35.54m 35.93m ~ 0.400
_SubtreeDataDeserializationWithBufferSizes/128KB-4 35.80m 35.60m ~ 0.700
_SubtreeDataDeserializationWithBufferSizes/512KB-4 36.09m 35.21m ~ 0.100
_PooledVsNonPooled/Pooled-4 737.0n 738.4n ~ 0.700
_PooledVsNonPooled/NonPooled-4 7.203µ 7.081µ ~ 0.400
_MemoryFootprint/Current_512KB_32concurrent-4 7.397µ 7.105µ ~ 0.700
_MemoryFootprint/Proposed_32KB_32concurrent-4 10.68µ 10.83µ ~ 0.100
_MemoryFootprint/Alternative_64KB_32concurrent-4 10.12µ 10.35µ ~ 0.100
_prepareTxsPerLevel-4 414.8m 411.1m ~ 0.700
_prepareTxsPerLevelOrdered-4 3.567m 3.995m ~ 0.200
_prepareTxsPerLevel_Comparison/Original-4 423.9m 422.2m ~ 0.400
_prepareTxsPerLevel_Comparison/Optimized-4 3.942m 3.821m ~ 0.100
SubtreeSizes/10k_tx_4_per_subtree-4 1.434m 1.464m ~ 0.700
SubtreeSizes/10k_tx_16_per_subtree-4 334.6µ 345.9µ ~ 0.100
SubtreeSizes/10k_tx_64_per_subtree-4 80.02µ 81.47µ ~ 0.700
SubtreeSizes/10k_tx_256_per_subtree-4 19.80µ 20.06µ ~ 0.400
SubtreeSizes/10k_tx_512_per_subtree-4 9.990µ 10.013µ ~ 0.700
SubtreeSizes/10k_tx_1024_per_subtree-4 4.948µ 4.948µ ~ 1.000
SubtreeSizes/10k_tx_2k_per_subtree-4 2.439µ 2.435µ ~ 0.400
BlockSizeScaling/10k_tx_64_per_subtree-4 79.09µ 77.34µ ~ 0.100
BlockSizeScaling/10k_tx_256_per_subtree-4 19.65µ 19.70µ ~ 1.000
BlockSizeScaling/10k_tx_1024_per_subtree-4 4.886µ 4.893µ ~ 1.000
BlockSizeScaling/50k_tx_64_per_subtree-4 409.4µ 407.2µ ~ 0.700
BlockSizeScaling/50k_tx_256_per_subtree-4 98.59µ 97.89µ ~ 0.100
BlockSizeScaling/50k_tx_1024_per_subtree-4 24.12µ 24.26µ ~ 0.700
SubtreeAllocations/small_subtrees_exists_check-4 166.3µ 163.3µ ~ 0.200
SubtreeAllocations/small_subtrees_data_fetch-4 177.8µ 172.0µ ~ 0.400
SubtreeAllocations/small_subtrees_full_validation-4 341.4µ 339.9µ ~ 0.700
SubtreeAllocations/medium_subtrees_exists_check-4 9.868µ 9.754µ ~ 0.400
SubtreeAllocations/medium_subtrees_data_fetch-4 10.32µ 10.29µ ~ 1.000
SubtreeAllocations/medium_subtrees_full_validation-4 20.02µ 19.73µ ~ 0.100
SubtreeAllocations/large_subtrees_exists_check-4 2.376µ 2.331µ ~ 0.200
SubtreeAllocations/large_subtrees_data_fetch-4 2.501µ 2.462µ ~ 0.100
SubtreeAllocations/large_subtrees_full_validation-4 5.003µ 4.946µ ~ 0.400
StoreBlock_Sequential/BelowCSVHeight-4 319.1µ 317.1µ ~ 0.200
StoreBlock_Sequential/AboveCSVHeight-4 321.5µ 330.4µ ~ 1.000
GetUtxoHashes-4 257.4n 257.4n ~ 1.000
GetUtxoHashes_ManyOutputs-4 43.90µ 43.53µ ~ 0.100
_NewMetaDataFromBytes-4 242.7n 240.3n ~ 0.100
_Bytes-4 625.9n 624.7n ~ 1.000
_MetaBytes-4 575.3n 563.7n ~ 0.300

Threshold: >10% with p < 0.05 | Generated: 2026-05-12 17:26 UTC

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a legacy netsync quick-validation regression where createUtxos could silently skip applying the current block ID to transactions that already existed in the UTXO store (returned ErrTxExists). That left those txs with empty/stale BlockIDs, causing descendant blocks to fail validation (“parent transaction … has no block IDs”) and triggering cascading invalidation.

Changes:

  • In createUtxos, collect tx hashes that return ErrTxExists during parallel Create calls.
  • After the parallel create pass completes, call SetMinedMulti once to merge the current block’s mined info into those pre-existing tx records.
  • Add a unit test that pre-creates a tx with empty BlockIDs, runs createUtxos, and asserts the block ID was merged.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
services/legacy/netsync/handle_block.go Tracks ErrTxExists txs during UTXO creation and batch-merges the blockID via SetMinedMulti to prevent empty/stale BlockIDs.
services/legacy/netsync/handle_block_test.go Adds a targeted regression test ensuring createUtxos merges the block ID for pre-existing tx records.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@oskarszoon oskarszoon requested review from freemans13 and ordishs May 12, 2026 19:04
@oskarszoon oskarszoon merged commit efaeedf into bsv-blockchain:main May 13, 2026
29 checks passed
oskarszoon added a commit that referenced this pull request May 24, 2026
createUtxos was calling SetMinedMulti with one unbounded slice containing every
pre-existing tx in the block. On fat mainnet blocks (e.g. 755880 = 2.87M txs)
this monolithic batch exhausts the aerospike client connection pool and stalls
sync in a MAX_RETRIES_EXCEEDED retry loop. Regression from #854.

Split the merge across a worker pool bounded by UtxoStore.MaxMinedRoutines,
each worker iterating its range in MaxMinedBatchSize chunks. Mirrors the
proven MarkTransactionsOnLongestChain pattern.

Fail-fast errgroup semantics match the existing createUtxos Create() loop:
first chunk error cancels siblings via gCtx and propagates a wrapped
ProcessingError.

Fixes #936
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants