fix(asset): address review feedback from #827 (streaming follow-ups)#880
Conversation
- subtree page (json): return empty page when offset is past totalNodes instead of silently wrapping to 0; aligns with readSubtreeNodesPageFromReader and avoids surprising paginated clients. - settings.conf: document the actual non-zero defaults for the asset subtree concurrency keys (2 / 4 / 2 / 2) so operators don't believe the documented "0 = unlimited" comment after these were silently bounded. - bench: add an un-gated BenchmarkSubtreeNodeHashesReader plus a regression test that runs it via testing.Benchmark and asserts AllocsPerOp <= 16, so a future regression to non-streaming buffering fails CI. - GetLegacyBlock: in scheduleSubtreeChunkFetches, prefer waitErr (the real worker error) over readErr (which is the context.Canceled propagated when the errgroup cancelled gCtx). Falls back to readErr when waitErr is nil or itself a cancellation, so genuine read errors are still surfaced. - GetSubtree handler: collapse the trailing dead switch + unreachable return nil into an if/else (mode is already validated above). - docs: assetServer.md gains a Deployment section documenting that the gzip-skipped /subtree* endpoints require upstream rate-limiting / ACL in a reverse proxy, since per-IP rate limiting is not yet implemented in the asset service. - tests: targeted coverage for the new subtree_stream paths (negative args, offset past end, zero limit, page truncation, empty subtree, truncated header / conflict count, zero-length Read, context cancel, source.Close on header failure), plus micro-tests for the new drainChunkResults and sendChunkResult helpers. Notes for reviewers: - ConflictingNodes pagination semantics: partial pages return empty ConflictingNodes; only a page that covers the whole subtree (offset==0 && pageSize==totalNodes) reads them. This is the change introduced in b08a88e and now documented here. - FeeHash zero-value in /subtree/:hash/json is unchanged from prior behaviour; no regression.
|
🤖 Claude Code Review Status: Complete Current Review: This PR successfully addresses review feedback from #827. The changes are well-structured with appropriate tests and documentation. No issues found. Key improvements verified:
Documentation accuracy: All documentation changes accurately reflect the implementation. The |
There was a problem hiding this comment.
Pull request overview
This PR is a follow-up to #827 that folds in outstanding review feedback around asset subtree streaming semantics, configuration documentation, error propagation, and test verification of the streaming allocation claims.
Changes:
- Adjust subtree JSON paging to return an empty page when
offset >= totalNodes(instead of wrapping to offset 0) and clarifyConflictingNodespopulation semantics for full-subtree pages only. - Improve
scheduleSubtreeChunkFetcheserror reporting by preferring the actual worker error overcontext.Canceledcaused by errgroup cancellation. - Add always-on benchmark + alloc regression test coverage for the streaming node-hashes reader, plus additional unit tests for subtree streaming helpers/edge-cases; update docs/settings to reflect bounded defaults and operational guidance.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
settings.conf |
Documents the actual non-zero defaults for subtree-related asset concurrency settings and fixes formatting. |
services/asset/repository/subtree_stream.go |
Updates subtree JSON paging behavior to return empty pages past end and adjusts when conflicting nodes are read. |
services/asset/repository/subtree_stream_test.go |
Adds extensive unit tests plus a benchmark and an allocation regression test for the streaming reader. |
services/asset/repository/repository_test.go |
Updates repository-level test expectations for the new “offset past end returns empty page” behavior. |
services/asset/repository/GetLegacyBlock.go |
Ensures worker errors are surfaced instead of masking them with context.Canceled. |
services/asset/repository/GetLegacyBlock_test.go |
Adds micro-tests for drainChunkResults and sendChunkResult. |
services/asset/httpimpl/GetSubtree.go |
Simplifies the handler control flow by removing an unreachable/dead switch tail. |
docs/topics/services/assetServer.md |
Adds deployment guidance requiring upstream rate limiting/ACLs for large streaming subtree endpoints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Benchmark Comparison ReportBaseline: Current: Summary
All benchmark results (sec/op)
Threshold: >10% with p < 0.05 | Generated: 2026-05-15 16:37 UTC |
oskarszoon
left a comment
There was a problem hiding this comment.
All six items from my #827 review addressed at HEAD.
- Empty-page consistency: both readers share
limit == 0 || offset >= totalNodesearly-exit. settings.confconcurrency keys documented2/4/2/2.TestSubtreeNodeHashesReaderAllocsBoundruns unconditionally; 16-alloc ceiling meaningful (per-node regression would push AllocsPerOp to ~1024).scheduleSubtreeChunkFetchesprecedence: non-cancellationwaitErr→readErr→ nil. Correct, no deadlock.GetSubtreedead-switch collapsed; mode validation intact.assetServer.mdnames correct nginx directives (limit_req_zone,limit_conn_zone).
Items deliberately not addressed (rationale verified):
- FeeHash JSON was always zero-value; not a regression.
- CodeQL :118/:178 — two independent clamps (HTTP layer + repository
subtreePageMaxNodes) confirm the false-positive rationale. - claude[bot] http.go:435 —
parts[i+1:]cannot panic (slice tolenreturns empty).
Local: 696 tests pass, race-clean, vet/lint clean.



Follow-up to #827 (which was merged before all review feedback could be folded in).
Addresses comments from @oskarszoon and @ordishs.
Changes
readSubtreePageFromReadernow returns an empty page when the requested offset is past the end of the subtree (matchingreadSubtreeNodesPageFromReader) instead of silently wrapping back to offset 0. The wrap was confusing for paginated clients. (review item [FEAT] Peer service should sanity checkasset_httpPublicAddress#2)settings.conf— Documented the actual non-zero defaults for the asset subtree concurrency keys (asset_concurrency_get_subtree_data=2,..._data_reader=4,..._transactions=2,asset_subtreeDataStreamingConcurrency=2). The previous block claimed0 = unlimiteddefaults, which silently became2 / 4 / 2 / 2in Stream asset subtree responses to reduce memory use #827. (review item Why can mining nodes run without downloading the entire blockchain? #3)BenchmarkSubtreeNodeHashesReaderand a regression test (TestSubtreeNodeHashesReaderAllocsBound) that runs it viatesting.Benchmarkand assertsAllocsPerOp <= 16. The original benchmarks were behindRUN_SUBTREE_BENCHMARK=trueand never ran in CI, so the streaming memory savings claim was not machine-verified. (review item Is it possible to develop a mining pool for the testnet using Teranode? #4)scheduleSubtreeChunkFetcheserror masking — PreferwaitErr(real worker error) overreadErr(which is thecontext.Canceledwe observe when the errgroup cancelsgCtx). Falls back toreadErrwhenwaitErris nil or itself a cancellation, so genuine read errors are still surfaced. (review item fix(model): correct BlockHeaderMeta serialization and add ProcessedAt support #6)GetSubtreehandler — Collapsed the trailing deadswitch+ unreachablereturn nilinto a directif/else; mode is already validated above. (review item Is it possible to develop a mining pool for the testnet using Teranode? #4)assetServer.mdgains a section explaining that the gzip-skipped/subtree*endpoints require upstream rate-limiting / ACL in a reverse proxy (nginxlimit_req_zone/limit_conn_zoneor equivalent). The asset service does not yet do per-IP rate limiting; that is a planned defense-in-depth follow-up. (review item ops note)ConflictingNodespaginated-response semantics (documented here) —ConflictingNodesis now only populated when the page covers the whole subtree (offset==0 && pageSize==totalNodes). Partial pages return an emptyConflictingNodesarray. This semantic was introduced in 827'sb08a88e3to avoid scanning all 1M nodes when paginating a large subtree; calling it out explicitly so any downstream consumer can adapt. (review item Why can mining nodes run without downloading the entire blockchain? #3)subtree_stream.gopaths (negative args, offset past end, zero limit, page truncation, empty subtree, truncated header / conflict count, zero-lengthRead, context cancel,source.Closeon header read failure), plus micro-tests fordrainChunkResultsandsendChunkResult. Boostssubtree_stream.gofunction coverage to ~75–82% and clears the SonarQube new-code-coverage gap for these files. (ordishs review item 1, 4)Items not addressed here
FeeHashJSON value — Confirmed not a regression; old behaviour also emitted zero-valueFeeHashin/subtree/:hash/json. No code change needed. (review item #1)subtree_stream.go:118and:178— False positives;pageSizeis provably bounded to ≤ 100 by both the HTTP-layer clamp (helpers.go:107-109) and the repository-layer defense-in-depth clamps (subtree_stream.go:86-88, 142-144). Max allocation ~4.8 KB per request. Suggest dismissing in the Security tab with that rationale rather than adding dead defensive code.http.go:435— Already refuted in the original PR review (parts[i+1:]cannot panic; Go allows slicing tolen, and thelen(tail)==1/2guards preventtail[1]OOB).Test plan
go test -race ./services/asset/...passes locallygo vet ./services/asset/...cleanTestSubtreeNodeHashesReaderAllocsBoundpasses (locks in the streaming memory claim)make test— all asset-related packages pass; remaining failures are unrelated testcontainer (Docker daemon) issues inutil/kafkaandutil/usql