Describe the bug
Merkle proofs / BUMPs served by the Assets API (GET /api/v1/merkle_proof/:hash) are invalid for transactions in the first subtree of a block, and for all transactions in any multi‑subtree block.
Reported in discussion #989: for the first non‑coinbase transaction of a block the proof's level‑0 sibling is the literal coinbase placeholder ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff instead of the real coinbase txid, and the reconstructed merkle root does not match the block header merkle root.
Investigation confirmed the bug and found it is actually three distinct defects on the same ConstructMerkleProof → ConvertToBUMP code path.
Root cause
The first subtree of a block stores a coinbase placeholder at Nodes[0] (subtree.CoinbasePlaceholderHashValue, 32 × 0xFF), because the coinbase txid isn't known until the block is assembled. The block header merkle root is computed by replacing that placeholder with the real coinbase txid:
model/Block.go:1377 — CheckMerkleRoot → subtree.RootHashWithReplaceRootNode(b.CoinbaseTx.TxIDChainHash(), 0, …) for subtree 0, and RootHashPadded for an incomplete final subtree.
model/Block.go:516 — asserts SubtreeSlices[0].Nodes[0] is the placeholder.
But util/merkleproof/merkle_proof.go ConstructMerkleProof built proofs from the stored subtree with the placeholder still in place, so the proof could never reconstruct the header root:
- Coinbase placeholder not replaced.
subtreeData.GetMerkleProof(txIndexInSubtree) returns Nodes[0] (the placeholder) as the sibling for a tx at index 1, and proof.SubtreeRoot / GenerateBlockMerkleProof use the placeholder‑based subtree‑0 root rather than the coinbase‑replaced root the header used.
- tx#1 (index 1): placeholder is its direct sibling →
ffff… literally appears in the proof.
- tx#2 (index 2): placeholder is buried in
H(node0,node1) → no ffff…, but still a wrong root.
- Missing subtree‑level flags. Only block‑level flags were populated, so
VerifyMerkleProof used the wrong sibling ordering for any odd‑index transaction. This only ever "passed" in tests for index‑0 transactions.
- Wrong BUMP offsets for multi‑subtree blocks. A BUMP is a single flat merkle tree, so offsets must be global, but
ConvertToBUMP (util/bump/format.go) numbered the subtree‑level and block‑level segments independently. For any block with more than one subtree this produces BUMPs that the go‑bc reference implementation rejects (we do not have a hash for this index at height: N).
Additionally, an incomplete final subtree was not lifted (RootHashPadded) the way CheckMerkleRoot lifts it, so block‑level proofs diverged from the header root for multi‑subtree blocks whose last subtree is incomplete.
Why existing tests missed it
util/merkleproof/merkle_proof_test.go builds the first subtree with a fake real coinbase hash at Nodes[0] and derives the test's header merkle root from that same node — internally self‑consistent, but not how production stores subtrees. No test used the actual CoinbasePlaceholderHashValue, and the verifying tests only covered index‑0 / single‑subtree cases.
To Reproduce
- Query the Assets API for a merkle proof of the first non‑coinbase transaction in a block:
GET /api/v1/merkle_proof/{txid}/json.
- Observe the level‑0 sibling is
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff.
- Verify the BUMP (e.g.
go-bc CalculateRootGivenTxid) → the computed merkle root does not match the block header merkle root.
Reported reproductions (from #989):
| Network |
Block height |
txid |
Result |
| testnet |
1500074 |
aeee8aee575a1f582ed31bf661c7d8f95a3db920e36c0047df5e29394fb3f58c |
computed root ≠ expected |
| mainnet |
951239 |
f9dfdb024b9d917a1fb2b8983e031061aa0f77f157f3f50f8e9913f580d8e615 |
computed root ≠ expected |
Expected behavior
The served merkle proof / BUMP must reconstruct to the block header merkle root for every transaction (coinbase, first‑subtree, multi‑subtree, and incomplete‑final‑subtree), and must never contain the coinbase placeholder.
Affected code
util/merkleproof/merkle_proof.go — ConstructMerkleProof
util/bump/format.go — ConvertToBUMP
- consumed by
services/asset/httpimpl/GetMerkleProof.go
Note: the separate coinbase‑BUMP helper util/bump.ComputeCoinbaseBUMP already handles the placeholder correctly; only the ConstructMerkleProof path used by the Assets API is affected.
Proposed fix
ConstructMerkleProof should mirror model.Block.CheckMerkleRoot:
- Replace the coinbase placeholder in subtree 0 with the real coinbase txid for the subtree‑internal proof, the block‑level leaf, and
SubtreeRoot (work on Subtree.Duplicate() clones so stored subtrees are never mutated).
- Lift an incomplete final subtree with
RootHashPadded for the block‑level leaf, and append self‑hash padding levels when the proven tx lives in that subtree.
- Compute subtree‑level left/right flags from the tx index parity.
- Handle a request for the coinbase txid itself (stored as the placeholder) as index 0 of the first subtree.
ConvertToBUMP should emit a single continuous global offset space: globalOffset = (subtreeIndex << subtreeLevels) | txIndexInSubtree, with sibling offsets (globalOffset >> level) ^ 1 across both subtree and block levels.
Verification should include go‑bc cross‑checks (CalculateRootGivenTxid == header merkle root) for single‑subtree, multi‑subtree, and incomplete‑final‑subtree cases, plus regression tests that build the first subtree the production way (real CoinbasePlaceholderHashValue at Nodes[0]).
Additional context
Originally reported in discussion #989.
Describe the bug
Merkle proofs / BUMPs served by the Assets API (
GET /api/v1/merkle_proof/:hash) are invalid for transactions in the first subtree of a block, and for all transactions in any multi‑subtree block.Reported in discussion #989: for the first non‑coinbase transaction of a block the proof's level‑0 sibling is the literal coinbase placeholder
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffinstead of the real coinbase txid, and the reconstructed merkle root does not match the block header merkle root.Investigation confirmed the bug and found it is actually three distinct defects on the same
ConstructMerkleProof → ConvertToBUMPcode path.Root cause
The first subtree of a block stores a coinbase placeholder at
Nodes[0](subtree.CoinbasePlaceholderHashValue, 32 ×0xFF), because the coinbase txid isn't known until the block is assembled. The block header merkle root is computed by replacing that placeholder with the real coinbase txid:model/Block.go:1377—CheckMerkleRoot→subtree.RootHashWithReplaceRootNode(b.CoinbaseTx.TxIDChainHash(), 0, …)for subtree 0, andRootHashPaddedfor an incomplete final subtree.model/Block.go:516— assertsSubtreeSlices[0].Nodes[0]is the placeholder.But
util/merkleproof/merkle_proof.goConstructMerkleProofbuilt proofs from the stored subtree with the placeholder still in place, so the proof could never reconstruct the header root:subtreeData.GetMerkleProof(txIndexInSubtree)returnsNodes[0](the placeholder) as the sibling for a tx at index 1, andproof.SubtreeRoot/GenerateBlockMerkleProofuse the placeholder‑based subtree‑0 root rather than the coinbase‑replaced root the header used.ffff…literally appears in the proof.H(node0,node1)→ noffff…, but still a wrong root.VerifyMerkleProofused the wrong sibling ordering for any odd‑index transaction. This only ever "passed" in tests for index‑0 transactions.ConvertToBUMP(util/bump/format.go) numbered the subtree‑level and block‑level segments independently. For any block with more than one subtree this produces BUMPs that the go‑bc reference implementation rejects (we do not have a hash for this index at height: N).Additionally, an incomplete final subtree was not lifted (
RootHashPadded) the wayCheckMerkleRootlifts it, so block‑level proofs diverged from the header root for multi‑subtree blocks whose last subtree is incomplete.Why existing tests missed it
util/merkleproof/merkle_proof_test.gobuilds the first subtree with a fake real coinbase hash atNodes[0]and derives the test's header merkle root from that same node — internally self‑consistent, but not how production stores subtrees. No test used the actualCoinbasePlaceholderHashValue, and the verifying tests only covered index‑0 / single‑subtree cases.To Reproduce
GET /api/v1/merkle_proof/{txid}/json.ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff.go-bcCalculateRootGivenTxid) → the computed merkle root does not match the block header merkle root.Reported reproductions (from #989):
aeee8aee575a1f582ed31bf661c7d8f95a3db920e36c0047df5e29394fb3f58cf9dfdb024b9d917a1fb2b8983e031061aa0f77f157f3f50f8e9913f580d8e615Expected behavior
The served merkle proof / BUMP must reconstruct to the block header merkle root for every transaction (coinbase, first‑subtree, multi‑subtree, and incomplete‑final‑subtree), and must never contain the coinbase placeholder.
Affected code
util/merkleproof/merkle_proof.go—ConstructMerkleProofutil/bump/format.go—ConvertToBUMPservices/asset/httpimpl/GetMerkleProof.goProposed fix
ConstructMerkleProofshould mirrormodel.Block.CheckMerkleRoot:SubtreeRoot(work onSubtree.Duplicate()clones so stored subtrees are never mutated).RootHashPaddedfor the block‑level leaf, and append self‑hash padding levels when the proven tx lives in that subtree.ConvertToBUMPshould emit a single continuous global offset space:globalOffset = (subtreeIndex << subtreeLevels) | txIndexInSubtree, with sibling offsets(globalOffset >> level) ^ 1across both subtree and block levels.Verification should include go‑bc cross‑checks (
CalculateRootGivenTxid == header merkle root) for single‑subtree, multi‑subtree, and incomplete‑final‑subtree cases, plus regression tests that build the first subtree the production way (realCoinbasePlaceholderHashValueatNodes[0]).Additional context
Originally reported in discussion #989.