feat(asset/http): POST /api/v1/utxos bulk lookup + drop redundant tx fetch on GetUTXO#950
Conversation
…fetch on GetUTXO Addresses bsv-blockchain#945. The single-UTXO endpoint was fetching the full raw transaction per request just to recompute a hash the store already had, which dominated latency and made parallel client requests no faster than sequential ones. Three coordinated changes: 1. stores/utxo/{aerospike,sql}: make GetSpend tolerate spend.UTXOHash == nil. Both stores locate the record by primary key (txid, vout) and already know the canonical hash; the caller-supplied hash was a defensive check. Existing callers still pass a hash; only the new HTTP fast paths opt out. The SQL path also stops dereferencing nil. 2. services/asset/httpimpl/GetUTXO.go: stop calling GetTransaction + NewTxFromBytes + UTXOHashFromOutput. Each request is now one Aerospike record read instead of (blob-fetch + tx-parse + Aerospike Get). The "output index out of range" path collapses into the existing Status_NOT_FOUND branch; error message is now uniform "UTXO not found". 3. services/asset/httpimpl/GetUTXOs.go (new): POST /api/v1/utxos. Binary request body of 36-byte (32B txid + 4B vout LE) records, binary response of fixed 48-byte records (status + lockTime + spendingVin + spendingTxID) in input order, zero-padded when unspent so clients can index by i*48. Fan-out via errgroup with SafeSetLimit(1024), preallocated response with per-slot writes (no mutex). Body size capped by a new asset_httpBodyLimit setting (default 4MB, ~116k records). Tests: - new TestGetSpendNilUTXOHash (SQL): nil skips the check, wrong non-nil still fails. - TestGetUTXO updated for the new path; new "Does not call GetTransaction" regression subtest. - new TestGetUTXOs: empty body, malformed length, mixed found/unspent/spent/not-found with order preservation, UTXOHash nil assertion, per-record vs whole-request error policy.
|
🤖 Claude Code Review Status: Complete Current Review:
Summary:
The implementation is solid, well-tested, and follows the project verification workflow from AGENTS.md. Documentation accurately describes the new bulk endpoint behavior including the known lockTime semantic divergence between SQL and Aerospike backends. |
|
[Minor] Error format mismatch in GetUTXOs.go line 142 The format string "error getting utxo %s:%d" has 2 placeholders but 3 arguments are provided (txHash.String(), vout, err). The error will be wrapped but not shown in the message. Suggested fix: This matches the pattern used elsewhere in the codebase (see stores/utxo/aerospike/spend.go:194 for similar usage). |
There was a problem hiding this comment.
Pull request overview
This PR adds a bulk UTXO spend-status lookup endpoint to the Asset HTTP service and optimizes the existing single-UTXO endpoint by removing redundant transaction fetch/parse work, relying on the UTXO store to locate records by (txid, vout).
Changes:
- Add
POST /api/v1/utxosbulk binary lookup endpoint with bounded fan-out, fixed-size binary responses, and Prometheus tracking. - Speed up
GET /api/v1/utxo/:hash?vout=Nby skippingGetTransaction+ tx parsing + recomputed UTXO hash. - Allow
GetSpendto acceptspend.UTXOHash == nilin both SQL and Aerospike stores; add a configurableasset_httpBodyLimit.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| stores/utxo/sql/sql.go | Allow nil Spend.UTXOHash by skipping the mismatch check when absent. |
| stores/utxo/sql/sql_test.go | Add coverage for nil-hash GetSpend behavior and mismatch behavior when non-nil. |
| stores/utxo/aerospike/get.go | Allow nil Spend.UTXOHash by skipping the mismatch check when absent. |
| settings/settings.go | Wire new asset_httpBodyLimit into constructed settings. |
| settings/asset_settings.go | Add AssetSettings.HTTPBodyLimit metadata for the new setting. |
| services/asset/httpimpl/metrics.go | Add Prometheus counter for bulk UTXO requests. |
| services/asset/httpimpl/http.go | Register POST /api/v1/utxos route with optional per-route BodyLimit middleware. |
| services/asset/httpimpl/GetUTXOs.go | Implement bulk binary request/response handler with concurrency cap and per-slot writes. |
| services/asset/httpimpl/GetUTXOs_test.go | Add handler tests for ordering, error mapping, and fast-path behavior. |
| services/asset/httpimpl/GetUTXO.go | Remove transaction fetch/parse; call GetUtxo with UTXOHash: nil. |
| services/asset/httpimpl/GetUTXO_test.go | Update tests for unified NOT_FOUND message and ensure GetTransaction isn’t called. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark Comparison ReportBaseline: Current: Summary
All benchmark results (sec/op)
Threshold: >10% with p < 0.05 | Generated: 2026-05-27 16:55 UTC |
…1/utxos Address review feedback on PR bsv-blockchain#950. Critical fix: - services/asset/repository/repository.go:917 dereferenced spend.UTXOHash for a debug log line. The new POST /api/v1/utxos handler (and the simplified GET /api/v1/utxo path) pass UTXOHash: nil, which would have panicked here at runtime. Log TxID:Vout instead — both fields are always populated. - New TestGetUtxoNilUTXOHash exercises the real Repository (not the mock) against a sqlitememory store so the panic would surface in CI. The handler-layer tests used asset/repository.Mock which doesn't traverse this code path, which is why the bug slipped through the first review. Documentation: - swagger_routes.go: add the missing POST /api/v1/utxos entry. - docs/references/services/asset_reference.md: document the new endpoint (method, request/response layout, body cap, error responses). - docs/topics/services/img/plantuml/assetserver/asset_server_http_get_utxo.{puml,svg}: extend the sequence diagram with the bulk lookup flow. - settings/asset_settings.go: reword asset_httpBodyLimit desc/longdesc to make clear it scopes only to /utxos, not the whole Asset HTTP surface. - services/asset/httpimpl/http.go: matching comment tweak. Cleanups: - services/asset/httpimpl/GetUTXO_test.go: strip seven dead "tx := bt.NewTx() + GetTransaction mock" setups left over from the removed code path; drop the now-unused go-bt import; switch back to "err :=" where the declaration was removed. - services/asset/httpimpl/GetUTXOs.go: drop the strings.Contains("not found") fallback (errors.Is(err, errors.ErrNotFound) suffices and is consistent with the always-use-teranode/errors convention); tighten the response slot slice to its exact 48-byte bounds.
… modes
Mirror the GET /api/v1/utxo {BINARY_STREAM, HEX, JSON} pattern on the bulk
endpoint. All three routes accept the same 36-byte binary request body; only
the response format differs.
- POST /api/v1/utxos — binary (fixed 48-byte records, indexable by i*48)
- POST /api/v1/utxos/hex — same binary blob, lowercase-hex encoded
- POST /api/v1/utxos/json — JSON array of utxo.SpendResponse in input order
GetUTXOs now takes a ReadMode parameter, the goroutines fill a
[]*utxo.SpendResponse slice (preserving input order via slot writes), and a
final writeUTXOsResponse helper serializes per mode. The binary record layout
is unchanged.
- swagger_routes.go: annotations for the two new routes.
- docs/references/services/asset_reference.md: documents all three modes.
- docs/topics/.../asset_server_http_get_utxo.{puml,svg}: diagram updated.
- New tests: HEX mode (hex-decode + record-layout assertions), JSON mode
(array shape + per-record fields + order), JSON mode on empty body (empty array).
oskarszoon
left a comment
There was a problem hiding this comment.
Approve with discussion items. 573/573 -race clean. Settings loader correctly wired this time (settings.go:215 reads asset_httpBodyLimit — #933 / #643 P1.1 pattern doesn't repeat).
Worth confirming before merge
-
Aerospike
GetSpendnever populatesLockTime(stores/utxo/aerospike/get.go:272-275). SQL path does viacoinbaseSpendingHeight. Binary/HEX response's lockTime is always 0 on Aerospike backend. Pre-existing, but bulk endpoint makes it machine-visible. Intentional → document in swagger; oversight → one-line fix. -
Endpoint anonymous + unrated.
apiGrouponly wires Recover + BanList + CORS + Gzip + securityHeaders. PR #643 (auth + rate limit) still CHANGES_REQUESTED. Single 4MB POST = ~116k AerospikeGets at 1024 concurrency through the shareduaerospike.Clientsemaphore (~256) that validator + block validation + propagation also use. Per-request cost is ~116k× the old single-UTXO endpoint. Worth a tracking issue for gating underasset_httpHeavyRateLimitonce #643 lands. -
Dropped
UTXOHashcheck was the only run-time detector for writer-sidevout % batchSizeslot-indexing regressions in Aerospike. Future writer bug would silently return wrong data. Add writer-side regression test (GetSpend(vout=X, UTXOHash=nil)afterSpend(vout=Y)with batchSize > 1) or hash-mismatch metric. -
Doc gaps:
asset_httpBodyLimitmissing fromdocs/references/settings/services/asset_settings.md— same shape as #643 v3.asset_reference.md:771-774doesn't call out the message-collapse behaviour change ("tx not found" + "vout out of range" → "NOT_FOUND (3): UTXO not found"). Scripted clients break silently.
Nits
GetUTXO.go:132loosestrings.Contains(err.Error(), "not found")whileGetUTXOs.go:149useserrors.Is(err, errors.ErrNotFound)— tighten singular.GetUTXOs_test.gomixesassertwith projectrequire-only convention.GetUTXO.go:37docstring calls:hash"UTXO hash"; it's the txid.- No 413 body-cap test —
GetMockHTTPbypasses BodyLimit middleware. c.Request().RemoteAddrin tracing log atGetUTXOs.go:92— wrong behind a reverse proxy, consistent with pre-existingGetUTXO.go:91.
errgroup fan-out at 1024 safe (per-slot writes, i := i capture, client disconnect cancels gCtx). Binary layout 8+4+4+32=48 verified. UTXOHash == nil correct in both stores; SQL latent nil-deref fixed at sql.go:3320. a3e578afb nil-safe logging verified end-to-end. Mode dispatch closure-bound at route registration — no header-tricking surface.
ReviewClean design overall — the binary layout is well-documented, the store nil- 🔴 CRITICAL — Unauthenticated DoS: out-of-range
|
| Path | Panic goroutine | middleware.Recover() covers it? |
Result |
|---|---|---|---|
GET /api/v1/utxo/:hash?vout=N |
echo request goroutine | ✅ yes (http.go:129) |
recovered → 500 (regression from the old clean 404) |
POST /api/v1/utxos |
errgroup child goroutine (GetUTXOs.go:524) |
❌ no — recover() is goroutine-local; errgroup does not recover |
unrecovered panic → asset process crashes |
Full chain for the bulk path has no recover anywhere: GetUTXOs goroutine → repository.GetUtxo (repository.go:911) → aerospike GetSpend (panics). Echo's recover middleware only protects the request goroutine, not the ones errgroup spawns.
Reproduction (unauthenticated, one request):
POST /api/v1/utxos
body = <32-byte txid of ANY tx with <128 outputs> || <uint32 LE vout in (numOutputs, 127]>
The coinbase of any block (1 output, well-known txid) with vout=1 suffices.
Required fix (root cause, protects all callers, minimal): bounds-check the offset in aerospike/get.go and treat overflow as not-found, mirroring the existing ErrKeyNotFound branch:
idx := spend.Vout % uint32(s.utxoBatchSize)
if int(idx) >= len(utxos) {
return &utxo.SpendResponse{Status: int(utxo.Status_NOT_FOUND)}, nil
}
b, ok := utxos[idx].([]byte)SQL is already safe (WHERE ... AND o.idx = $2 → ErrNoRows → Status_NOT_FOUND, sql.go:3303-3310), so the gap is aerospike-only (production).
Strongly recommend additionally isolating panics in the errgroup body so any future store-level panic degrades to 500 instead of taking down the process — a public fan-out endpoint should never let a per-record panic escape the goroutine.
Test gap that hid this: every GetUTXOs/GetUTXO test mocks the repository, so "Vout out of range" returns a mocked Status_NOT_FOUND and never exercises the real store indexing. Per the repo's testing rule (use sqlitememory, don't mock the store), an out-of-range-vout test against a real store would have caught the aerospike-vs-SQL divergence. Please add one.
🟡 Minor
413can be masked as500.GetUTXOsreads the body viaio.ReadAll(GetUTXOs.go:493). Echo'sBodyLimitreturns 413 up-front only via theContent-Lengthcheck; a client streaming without an accurateContent-Lengthtrips the limit insideRead, soio.ReadAllreturnsErrStatusRequestEntityTooLargeand the handler wraps it as500. Common clients sendContent-Lengthso the documented 413 holds for them — consider detectingecho.ErrStatusRequestEntityTooLargeand surfacing 413.- Behavior change (already flagged by author): "tx not found" vs "output index out of range" now collapse to
NOT_FOUND (3): UTXO not found. Documented, status code unchanged — just confirm no internal client greps the old message strings. - Error counter only on success.
prometheusAssetHTTPGetUTXOsis incremented only on the 200 path; 400/413/500 record nothing. Mirrors existingGetUTXO, so the bulk endpoint's failure rate will be invisible.
🟢 Nits
i := i(GetUTXOs.go:516) is dead since Go 1.22;go.modisgo 1.26.0.writeUTXOsRecorddoc says[...4 vin][32 txid]; PR body /asset_reference.mdsay "spendingVin"/"spendingTxID" — same thing, align wording.
Verdict
Request changes. Endpoint design, docs, and test breadth are strong. But removing the vout < len(outputs) guard exposes a pre-existing unchecked slice index in the aerospike store, and routing it through errgroup goroutines turns it into a one-request remote crash of the asset server.
Before merge:
- Bounds-check the offset in
aerospike/get.go(returnStatus_NOT_FOUND). - Add panic isolation around the
errgroupper-record body. - Add a real-store (sqlitememory + aerospike) out-of-range-vout test.
- (Optional) map
ErrStatusRequestEntityTooLargefromio.ReadAllto 413.
Address ordishs' critical finding on PR bsv-blockchain#950. Both new code paths (POST /api/v1/utxos, simplified GET /api/v1/utxo) forwarded a client-controlled vout straight to the store with no bounds check. The aerospike store indexed the per-batch utxo list without validating the offset — a vout greater than the actual output count panicked with index-out-of-range. In the bulk endpoint that panic escaped to an errgroup goroutine that echo.middleware.Recover does not cover, crashing the asset process from a single unauthenticated request. The bug was pre-existing in the store but previously unreachable: the old GetUTXO handler validated `vout < len(tx.Outputs)` against the parsed tx before calling GetSpend. The single-endpoint optimisation removed that guard; the bulk endpoint never had one. Fixes: - stores/utxo/aerospike/get.go: bounds-check `vout % batchSize` against the actual length of the Utxos bin; return Status_NOT_FOUND on overflow, mirroring the existing ErrKeyNotFound branch. SQL store already safe (WHERE ... AND o.idx = $2 -> ErrNoRows -> Status_NOT_FOUND). - services/asset/httpimpl/GetUTXOs.go: wrap each errgroup goroutine body in a deferred recover that converts any future store-level panic into a 500 errgroup error with a logged stack trace, so a public fan-out endpoint can never let a per-record panic escape the process. Regression guards: - stores/utxo/sql/sql_test.go: assert out-of-range vout returns Status_NOT_FOUND, not error/panic (locks the cross-store contract). - stores/utxo/aerospike/aerospike_server_test.go: same contract on the real aerospike store, in the existing TestAerospike subtree. - services/asset/httpimpl/GetUTXOs_test.go: panic-in-store test asserts the bulk handler returns 500 instead of crashing. Test-gap fix: every previous GetUTXOs/GetUTXO test mocked the repository, so out-of-range vout returned a mocked Status_NOT_FOUND and never exercised the real store indexing. The new SQL+aerospike tests close that gap. Other review items from this round: - services/asset/httpimpl/GetUTXOs.go: map echo.ErrStatusRequestEntityTooLarge from io.ReadAll to 413 so streaming clients without Content-Length get the documented status instead of 500. - Drop the now-redundant `i := i` (go.mod is go 1.26, loop-var fix landed in 1.22). - writeUTXOsRecord doc comment now uses spendingVin/spendingTxID to match public docs. - docs/references/services/asset_reference.md: document the pre-existing Aerospike/SQL LockTime divergence (Aerospike returns tx.LockTime; SQL returns coinbaseSpendingHeight). Bulk endpoint surfaces it more visibly but the gap pre-dates this PR.
|
Thanks for the careful review @ordishs — the panic chain is exactly as you described and the test gap is a fair critique. Pushed Critical fixes
Minor
Nits
Re @oskarszoon's items
Re the github-actions bot's "error format mismatch"The format string |
Re-review — all blocking feedback addressed ✅Re-reviewed at 🔴 Critical DoS — fixed at the root
idx := spend.Vout % uint32(s.utxoBatchSize)
if int(idx) >= len(utxos) {
return &utxo.SpendResponse{Status: int(utxo.Status_NOT_FOUND)}, nil
}
b, ok := utxos[idx].([]byte)Verified correct across single-batch, cross-batch, and exact-boundary vouts (e.g. 200-output tx: 🟢 Defense-in-depth — added
🟢 413 mapping — added and confirmed live
🟢 Test gap — closed at all three levels
The Verification I ran
Non-blocking
LGTM once CI is green. Nice, thorough turnaround. |
oskarszoon
left a comment
There was a problem hiding this comment.
Approve at 830258f260. DoS chain closed at both layers (defense-in-depth verified).
The critical
aerospike/get.go:192-197 bounds-check returns Status_NOT_FOUND before slice index — matches existing ErrKeyNotFound shape. GetUTXOs.go:141-156 errgroup per-goroutine defer recover() is the first statement, named return propagates panic→500 through g.Wait(). Surgical fix + general belt.
Tests cover all three layers:
sql_test.go:396-413— sqlitememory cross-store contract.aerospike_server_test.go:226-251— testcontainers, vout=99 on 2-output tx (panics pre-fix).GetUTXOs_test.go:255-281— handler-level panic→500.
All 673 tests -race clean.
Prior items addressed
- LockTime divergence (SQL coinbase maturity height vs Aerospike
nLockTime) documented inasset_reference.md:790. asset_httpBodyLimitcap + 413 mapping documented inasset_reference.md:788-789.- UTXOHash nil-skip design noted in both store code comments.
- 413 mapping for
echo.ErrStatusRequestEntityTooLargecleanly separates 4xx from 5xx — better ops alerting.
Follow-ups (not blocking)
- Operator-settings doc (
docs/references/settings/services/asset_settings.md) doesn't listasset_httpBodyLimityet — one-line add. - Recover handler emits a full stack trace per panicking record. Closed paths are safe; if a future new panic source ever slips through, 116k records × ~5KB ≈ ~580MB log output per malicious request. Sample first stack only.
- Anonymous + unrated endpoint stays on the follow-up list pending #643.
The panic-recovery defer in the bulk /api/v1/utxos goroutine logged the recovered value with `\n%s` + `debug.Stack()`, which embeds raw newlines and breaks log parsers. The project convention is single-line log messages. Match the existing pattern in services/propagation/Server.go: log just the panic value, txid, and vout on one line. Also document the panic-recovery 500 path in the handler doc comment so operators know that some 500s carry a logged-but-redacted panic value.
ordishs
left a comment
There was a problem hiding this comment.
Approving — CI is green, including the services/asset/httpimpl race run and the aerospike out-of-range-vout regression test.
The critical remote-panic DoS (out-of-range vout → index-out-of-range in the aerospike store, surfacing in an errgroup goroutine outside echo's recover) is fixed at the root via the batch-offset bounds check in aerospike/get.go, with defense-in-depth recover in the fan-out goroutines, the 413 mapping, and regression tests at the aerospike, SQL, and handler layers. All earlier feedback addressed. Nice turnaround.
Adopt PR bsv-blockchain#643's asset HTTP design: - Drop my per-route BodyLimit on POST /utxos in favour of the new top-level global middleware (asset_httpBodyLimit, default 100MB). - Add heavyMW()... to the three POST /utxos* routes so they are gated by the same tiered rate limiter as POST /subtree/:hash/txs. Closes the anonymous/unrated gap flagged in PR bsv-blockchain#950 review. - Drop the duplicate HTTPBodyLimit definition (settings/asset_settings.go) and the duplicate loader (settings/settings.go) — main's wins. - Update docs (asset_reference.md, GetUTXOs.go doc comment) for the new global cap, the rate limiter, and the 429 response code. Conflicts resolved: settings/asset_settings.go (kept main's HTTPBodyLimit definition wholesale).
… fix swagger doc Two PR bsv-blockchain#950 follow-ups from the latest bot runs: 1. swagger_routes.go:384 still said "default 4MB" but the post-merge cap is 100MB (the global asset_httpBodyLimit from PR bsv-blockchain#643). Aligned with asset_reference.md. 2. SonarCloud flagged the GetUTXOs handler with cognitive complexity 31 (limit 15). Extract two helpers — no behaviour change: - readUTXOsBody(c): read + 413 mapping + length-multiple validation. - fetchOneUTXO(ctx, txHash, vout): per-record store lookup with ErrNotFound / nil-resp / wrapped-error semantics. The panic-recover defer stays inline in the goroutine wrapper because it needs the closure variables. Handler now reads top-to-bottom: tracing → read body → fast-path empty → fan-out → wait → metrics → serialize. (The two MAJOR "filename does not follow snake_case" findings from Sonar are false positives — every Get*.go file under services/asset/httpimpl is CamelCase. Will flag in the PR reply.)
|
Addressed the latest bot/Sonar findings in 344d159. Fixed
Flagging as false positive (please confirm)
All 10 `TestGetUTXOs` subtests pass under `-race`; `make lint-new` clean. |
|




Addresses #945 — adds a bulk UTXO spend-status endpoint and fixes the underlying slowness that made the single-UTXO endpoint not benefit from client-side parallelism.
Summary
stores/utxo/{aerospike,sql}.GetSpend: toleratespend.UTXOHash == nil. The store already locates the record by primary key(txid, vout)and knows the canonical hash; the caller-supplied hash was a defensive check. Every existing caller still passes a hash, so the invariant is preserved for them — only the new HTTP fast paths opt out. The SQL path also stops dereferencing nil (was a latent panic).services/asset/httpimpl/GetUTXO.go: stop callingGetTransaction+NewTxFromBytes+UTXOHashFromOutput. EachGET /api/v1/utxo/:hash?vout=Nis now one Aerospike record read instead of (blob-fetch on cache miss) + (full tx parse) + (Aerospike Get). This is the latency the discussion was about.services/asset/httpimpl/GetUTXOs.go(new):POST /api/v1/utxos. Binary request body of 36-byte(txid || voutLE)records; binary response of fixed 48-byte records(statusLE || lockTimeLE || vinLE || spendingTxID)in input order, zero-padded when unspent so clients can index byi*48. Fan-out viaerrgroupcapped at 1024, preallocated response with per-slot writes (no mutex). Body size capped by a newasset_httpBodyLimitsetting (default 4MB, ≈116k records).Behaviour change to flag
The old single-endpoint distinguished "transaction not found" from "output index out of range" in the 404 message. Both now collapse to a uniform
NOT_FOUND (3): UTXO not found. The HTTP status is unchanged. Clients keying off the message string need to adjust; clients keying off status code do not.Test plan
go test -race ./services/asset/httpimpl/...(includes newTestGetUTXOsand updatedTestGetUTXO)go test -race ./stores/utxo/sql/...(includes newTestGetSpendNilUTXOHash; one pre-existing unrelated failureTestMinedThenSpendAllPrunes_*documented separately)go test ./stores/utxo/aerospike/...(full integration suite, exercises the nil-tolerance change)go vet,staticcheck,golangci-lintclean on touched packagesGetUTXOs.godoc comment)