Summary

Hotfix for production OOMKilled crashloops on the asset service caused by unbounded concurrent on-demand /subtree_data file creation.

The streaming work in #827 capped per-request memory but did not add admission control. Each dualStreamWithFileCreation call holds chunk-sized batches of full transaction metadata in memory; multiplied by however many clients arrive at once and by per-tx size variance (data carriers, multi-output txs), the asset process trivially exceeds even a 64Gi limit. This PR adds the missing admission cap, surfaces it as an HTTP 503 with Retry-After, and teaches the three known peer-side callers (subtreevalidation, blockvalidation catchup) to back off and retry instead of failing through.

Targets feat/teranode-native-ops because the dualStreamWithFileCreation path being protected lives on that branch (added via #826's cherry-pick chain).

Production evidence

The trigger for this work was the nginx-cache error stream:

upstream prematurely closed connection while reading response header from upstream,
client: 10.224.5.49, server: , request: "GET /api/v1/subtree_data/746a4271...",
upstream: "http://10.96.115.8:8090/api/v1/subtree_data/746a4271..."

Investigation on dev-scale-1-scale-1:

The "upstream prematurely closed connection" line is the only externally visible symptom because SIGKILL drops the TCP connection before any response headers can flush. Fixing the OOM eliminates it.

Root cause

Per-request memory was bounded by #827, but four amplifiers compound:

1. meta.Data.Tx is the full parsed bt.Tx. For OP_RETURN data carriers / multi-output txs it can run 10KB-500KB+. A 10000-tx chunk is 100MB-5GB, not the documented "~5MB".
2. dualStreamWithFileCreation writes through io.MultiWriter(storer, httpWriter). Both must accept each byte before the next chunk can write. Slow client OR slow file storer (Ensure atomic file store publication #826's atomic-rename path competing for global write semaphore) makes the producer block, and chunks pile up in flight + resultsChan + pending.
3. writeTransactionsViaSubtreeStoreStreaming is shared by GetSubtreeData, GetLegacyBlock, and mining_candidate_legacy_block — they fan out into the same getTxs machinery, multiplying goroutine pressure on Aerospike.
4. ConcurrencyGetSubtreeDataReader only caps the reader slot — it doesn't distinguish the cheap file-exists path from the memory-heavy on-demand creation path. With four expensive creations in flight you've already scheduled 50+ chunks worth of tx data in heap.

Changes

Asset — admission control

Util — typed 503 + retry helper

Callers — retry on peer 503

Three call sites switched from DoHTTPRequestBodyReader to DoHTTPRequestBodyReaderWithRetry:

• services/subtreevalidation/SubtreeValidation.go — getSubtreeMissingTxs
• services/subtreevalidation/check_block_subtrees.go — CheckBlockSubtrees
• services/blockvalidation/get_blocks.go — fetchSubtreeDataFromPeer (catchup)

Behavior changes — what clients should expect

Asset server

• Under nominal load: no observable change.
• Under create-path saturation: the 5th simultaneous on-demand creation gets HTTP 503 with Retry-After: 1 instead of waiting up to 30s for a permit (then 503'ing anyway via timeout). The pod stays up and serves all already-created files via the fast path.
• For an unhealthy cluster: a client that hammers the server during a stuck Aerospike batch gets 503s and is expected to retry. Far better failure mode than crashing.

Peer validation services

• subtreevalidation / blockvalidation: a peer's transient 503 is now retried (up to ~7.75s total worst case) instead of immediately falling through to "peer cannot provide subtree data" / next-peer attempt.
• This may slightly slow down detection of genuinely broken peers (a peer that always 503s now takes ~8s to give up), but eliminates the spurious "peer is broken" classification when the peer is just temporarily admission-throttled. Net positive for sync stability.

Test plan

Unit tests (in this PR)

• TestDoHTTPRequestBodyReaderWithRetry_SuccessOnFirstTry — no retry overhead when healthy
• TestDoHTTPRequestBodyReaderWithRetry_RetriesOn503ThenSucceeds — returns body of successful attempt, not 503 body
• TestDoHTTPRequestBodyReaderWithRetry_ExhaustsAttemptsOnPersistent503 — final error is typed ErrServiceUnavailable
• TestDoHTTPRequestBodyReaderWithRetry_HonorsRetryAfter — server Retry-After: 1 overrides much-smaller initialDelay
• TestDoHTTPRequestBodyReaderWithRetry_NoRetryOnNon503 (4 sub-cases) — 500/502/504/404 fail in 1 attempt
• TestDoHTTPRequestBodyReaderWithRetry_ContextCancelAbortsRetries — ctx cancel short-circuits the loop
• TestParseRetryAfter — empty/negative/non-numeric inputs return 0

Local verification done

• go build clean across util/, services/asset/, services/subtreevalidation/, services/blockvalidation/
• go vet clean (only pre-existing warnings in test/utils/ unrelated to this change)
• go test ./util/ -race passes in ~7s
• Pre-commit hooks: gci, gofmt, golangci-lint, etc. all green

Pending verification (cannot do locally)

• CI test suite (deferring to CI run on this PR)
• Deploy to dev-scale-1 and confirm:
  • asset pods stay up (no OOMKilled events)
  • 503s appear in metrics/logs under load instead of crashes
  • subtreevalidation / blockvalidation peers tolerate the 503s and complete catchup

Risks and rollout notes

• Feature flag-able via setting: asset_concurrency_subtree_data_create=0 reverts to unlimited (the prior behavior). Keep a knob in case the cap turns out to be too aggressive.
• The 503 path is new: clients of /subtree_data that don't go through the new retry helper will see 503s they didn't see before. The three known internal callers are updated; any external/unknown caller falls back to existing peer-failure handling, which already treats network errors as transient.
• Retry storm risk: 6 attempts × ~7.75s worst case per request × N peers could amplify load on a struggling asset server. Mitigated by the exponential backoff + Retry-After honoring + the 503-only filter (we don't retry on 5xx in general).
• No schema/wire changes: pure server-side and client-side error-handling change.
• Rollback: revert this commit. The deployed feat/teranode-native-ops branch returns to the prior (crashlooping) behavior — only do this if the new behavior is worse than the OOM, which would be surprising.

Companion / follow-up work (not in this PR)

• Lower default asset_subtreeDataStreamingChunkSize (currently 10000) — config-only change, can ship via Helm without a code change.
• Background quorum lock cleanup on startup — current per-request lazy expiration is fine but creates surprising latency when many stale locks exist after a crash storm.
• processTxMetaUsingStoreConcurrency review — getTxs fan-out is the largest goroutine multiplier; we may want a global cap rather than per-call.