Summary

Under catchup load — block-validator pulling many subtrees from a peer in parallel — the asset server logs an avalanche of `io: read/write on closed pipe` warnings. Measured rate on the scale-1 cluster: 4832 `write_failed` vs 82 `on_demand_created_locked` successes per ~1h window across 4 asset pods (~98% failure). Two coupled bugs, both with the same shape.

Bug 1 (consumer): cascading cancellation in both fan-outs

Two places fan out subtree_data fetches under `errgroup.WithContext`:

• `services/subtreevalidation/check_block_subtrees.go` (validation path)
• `services/blockvalidation/get_blocks.go` (catchup path)

When any one goroutine returns an error, `gCtx` is cancelled and every in-flight HTTP body `Read` returns `ctx.Err()`. Each client then closes its TCP connection. nginx (default `proxy_ignore_client_abort off`) propagates that close upstream. On the peer:

• `c.Response().Write` fails → `c.Stream` returns → `defer r.Close()` closes the pipe reader
• The producer goroutine's next write into `io.MultiWriter` returns `io.ErrClosedPipe`
• `storer.Abort(err)` discards already-paid Aerospike work

One failure → tens of cancelled streams across the pod fleet, every failed attempt wastes the chunked tx-meta fetch it already completed.

Fix: keep the errgroup but detach the heavy subtree_data work from `gCtx`. The existence check still respects the original context (so a pre-cancelled call exits fast); the HTTP fetch + parse + store does not. Sibling failures no longer cancel in-flight peers; each fetch completes naturally or hits its own `http_streaming_timeout`. The peer finishes writing its subtreeData file, which becomes a cache hit on the next retry. Trade-off: batch failure detection waits for in-flight peers to drain instead of bailing early — acceptable because the per-fetch streaming timeout still bounds it, and the alternative is throwing away pipeline work on every batch failure.

Bug 2 (producer): `write_failed` conflates client disconnects with server faults

`services/asset/repository/GetSubtreeData.go` logged every producer-side error at WARN under metric reason `write_failed`. `io.ErrClosedPipe` and context cancellation are caller-side conditions, not server faults — they should not pollute the ops signal that ought to flag genuine write failures.

Fix: detect `errors.IsContextError(err) || errors.Is(err, io.ErrClosedPipe)` and log at debug under a new metric reason `client_gone`. WARN + `write_failed` is now reserved for actual server-side write failures.

Files

• `services/subtreevalidation/check_block_subtrees.go` — pass `ctx` (not `gCtx`) to the subtree_data fetch + parser
• `services/blockvalidation/get_blocks.go` — detach `ctx` via `context.WithoutCancel` after the existence check in `fetchAndStoreSubtreeData`
• `services/asset/repository/GetSubtreeData.go` — classify `client_gone` separately from `write_failed`
• `services/asset/repository/metrics.go` — document the new `client_gone` reason
• `services/{asset/repository,subtreevalidation,blockvalidation}/*_test.go` — TDD regression tests

Tests

Written TDD-style — confirmed each test FAILS on pre-fix code and PASSES post-fix.

The cascade regression tests use a custom streaming response body that respects `req.Context()` between halves and waits for cancellation to propagate. httpmock's default buffered bodies cannot exercise the race because the full body is in memory by the time the reader runs; we needed an actual gated stream to prove that mid-stream cancellation truncates the body pre-fix and does not post-fix.

• `TestCheckBlockSubtrees_SiblingFailureDoesNotCancelInFlight` — subtree A succeeds streaming, sibling B fails fast; A's `FileTypeSubtreeData` file must be in the store afterwards.
• `TestFetchSubtreeDataForBlock_SiblingFailureDoesNotCancelInFlight` — same shape, catchup path.
• `TestGetSubtreeDataWithReader/client disconnect during stream is classified as client_gone, not write_failed` — producer-side classification.
• `TestGetSubtreeDataWithReader/genuine server-side write error still records write_failed` — companion test confirming WARN+`write_failed` is preserved for genuine faults (e.g. missing tx meta).

Out of scope (follow-ups)

• Settings-level mitigations (`asset_concurrency_subtree_data_create`, `asset_subtreeDataStreamingChunkSize`, nginx `proxy_ignore_client_abort on`) are operational tuning, separate concerns.
• The asset-side `storer.Abort` on `client_gone` still throws away the file in-progress. Decoupling the storer write from the response pipe (so a client disconnect lets the file finish to disk) is a bigger refactor and a separate PR.

Test plan

• `go test ./services/asset/repository/...` — pass
• `go test ./services/subtreevalidation/...` — pass
• `go test ./services/blockvalidation/...` — pass (incl. catchup subpackage)
• `go test -race ./services/asset/repository/...` — pass
• `go vet` on all three packages — clean
• `golangci-lint run --new-from-rev=origin/main` — clean
• Verify on scale-1 cluster: `teranode_asset_repository_subtree_data_created_total{result="error",reason="write_failed"}` rate drops to near-zero; `{reason="client_gone"}` absorbs the volume; `io: read/write on closed pipe` WARN lines disappear from the asset logs