Skip to content

[BUG] Assets API merkle proofs invalid: coinbase placeholder not replaced + wrong subtree flags + wrong multi-subtree BUMP offsets #992

@galt-tr

Description

@galt-tr

Describe the bug

Merkle proofs / BUMPs served by the Assets API (GET /api/v1/merkle_proof/:hash) are invalid for transactions in the first subtree of a block, and for all transactions in any multi‑subtree block.

Reported in discussion #989: for the first non‑coinbase transaction of a block the proof's level‑0 sibling is the literal coinbase placeholder ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff instead of the real coinbase txid, and the reconstructed merkle root does not match the block header merkle root.

Investigation confirmed the bug and found it is actually three distinct defects on the same ConstructMerkleProof → ConvertToBUMP code path.

Root cause

The first subtree of a block stores a coinbase placeholder at Nodes[0] (subtree.CoinbasePlaceholderHashValue, 32 × 0xFF), because the coinbase txid isn't known until the block is assembled. The block header merkle root is computed by replacing that placeholder with the real coinbase txid:

  • model/Block.go:1377CheckMerkleRootsubtree.RootHashWithReplaceRootNode(b.CoinbaseTx.TxIDChainHash(), 0, …) for subtree 0, and RootHashPadded for an incomplete final subtree.
  • model/Block.go:516 — asserts SubtreeSlices[0].Nodes[0] is the placeholder.

But util/merkleproof/merkle_proof.go ConstructMerkleProof built proofs from the stored subtree with the placeholder still in place, so the proof could never reconstruct the header root:

  1. Coinbase placeholder not replaced. subtreeData.GetMerkleProof(txIndexInSubtree) returns Nodes[0] (the placeholder) as the sibling for a tx at index 1, and proof.SubtreeRoot / GenerateBlockMerkleProof use the placeholder‑based subtree‑0 root rather than the coinbase‑replaced root the header used.
    • tx#1 (index 1): placeholder is its direct sibling → ffff… literally appears in the proof.
    • tx#2 (index 2): placeholder is buried in H(node0,node1) → no ffff…, but still a wrong root.
  2. Missing subtree‑level flags. Only block‑level flags were populated, so VerifyMerkleProof used the wrong sibling ordering for any odd‑index transaction. This only ever "passed" in tests for index‑0 transactions.
  3. Wrong BUMP offsets for multi‑subtree blocks. A BUMP is a single flat merkle tree, so offsets must be global, but ConvertToBUMP (util/bump/format.go) numbered the subtree‑level and block‑level segments independently. For any block with more than one subtree this produces BUMPs that the go‑bc reference implementation rejects (we do not have a hash for this index at height: N).

Additionally, an incomplete final subtree was not lifted (RootHashPadded) the way CheckMerkleRoot lifts it, so block‑level proofs diverged from the header root for multi‑subtree blocks whose last subtree is incomplete.

Why existing tests missed it

util/merkleproof/merkle_proof_test.go builds the first subtree with a fake real coinbase hash at Nodes[0] and derives the test's header merkle root from that same node — internally self‑consistent, but not how production stores subtrees. No test used the actual CoinbasePlaceholderHashValue, and the verifying tests only covered index‑0 / single‑subtree cases.

To Reproduce

  1. Query the Assets API for a merkle proof of the first non‑coinbase transaction in a block: GET /api/v1/merkle_proof/{txid}/json.
  2. Observe the level‑0 sibling is ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff.
  3. Verify the BUMP (e.g. go-bc CalculateRootGivenTxid) → the computed merkle root does not match the block header merkle root.

Reported reproductions (from #989):

Network Block height txid Result
testnet 1500074 aeee8aee575a1f582ed31bf661c7d8f95a3db920e36c0047df5e29394fb3f58c computed root ≠ expected
mainnet 951239 f9dfdb024b9d917a1fb2b8983e031061aa0f77f157f3f50f8e9913f580d8e615 computed root ≠ expected

Expected behavior

The served merkle proof / BUMP must reconstruct to the block header merkle root for every transaction (coinbase, first‑subtree, multi‑subtree, and incomplete‑final‑subtree), and must never contain the coinbase placeholder.

Affected code

  • util/merkleproof/merkle_proof.goConstructMerkleProof
  • util/bump/format.goConvertToBUMP
  • consumed by services/asset/httpimpl/GetMerkleProof.go

Note: the separate coinbase‑BUMP helper util/bump.ComputeCoinbaseBUMP already handles the placeholder correctly; only the ConstructMerkleProof path used by the Assets API is affected.

Proposed fix

ConstructMerkleProof should mirror model.Block.CheckMerkleRoot:

  1. Replace the coinbase placeholder in subtree 0 with the real coinbase txid for the subtree‑internal proof, the block‑level leaf, and SubtreeRoot (work on Subtree.Duplicate() clones so stored subtrees are never mutated).
  2. Lift an incomplete final subtree with RootHashPadded for the block‑level leaf, and append self‑hash padding levels when the proven tx lives in that subtree.
  3. Compute subtree‑level left/right flags from the tx index parity.
  4. Handle a request for the coinbase txid itself (stored as the placeholder) as index 0 of the first subtree.

ConvertToBUMP should emit a single continuous global offset space: globalOffset = (subtreeIndex << subtreeLevels) | txIndexInSubtree, with sibling offsets (globalOffset >> level) ^ 1 across both subtree and block levels.

Verification should include go‑bc cross‑checks (CalculateRootGivenTxid == header merkle root) for single‑subtree, multi‑subtree, and incomplete‑final‑subtree cases, plus regression tests that build the first subtree the production way (real CoinbasePlaceholderHashValue at Nodes[0]).

Additional context

Originally reported in discussion #989.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions