aerospike: optional native operate-path for mod-teranode UDFs#828
aerospike: optional native operate-path for mod-teranode UDFs#828icellan wants to merge 98 commits into
Conversation
Adds optional support for invoking mod-teranode functions via the native operate-path (TeranodeModifyOp, wire op type 200) instead of the legacy UDF path. The new path bypasses the Aerospike UDF executor and runs under the same lock as native ops like LIST_APPEND. Required server: the BSV fork of aerospike-server-private (feat/teranode-native-op or merged equivalent) on every node. Decision is wrapped at one place — call sites use s.teranodeBatchRecord(...) instead of aerospike.NewBatchUDF(...), and the helper picks the path based on: 1. Setting aerospike_use_native_teranode_ops (default false) 2. A startup capability probe against the cluster If either is false, calls fall back to UDF transparently. Clusters without the BSV fork keep working unchanged. Foundation: - go.mod: replace aerospike-client-go/v8 with the BSV fork - AerospikeSettings.UseNativeTeranodeOps (default false) - stores/utxo/aerospike/native_op.go: sub_op id constants, msgpack payload encoder, teranodeBatchRecord/executeTeranodeOp wrappers, capability probe (sends a malformed payload, expects PARAMETER_ERROR from the BSV dispatcher) - Store.useNativeTeranodeOps cached at construction Cutovers: - setMined Remaining cutovers (follow-up commits): preserveUntil, freeze, unfreeze, reassign, setConflicting, setLocked, spendMulti, unspend, incrementSpentExtraRecs (×2 sites). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Routes preserveUntil, freeze, unfreeze, reassign, setConflicting, setLocked, incrementSpentExtraRecs (×2), spendMulti, and unspend through the s.teranodeBatchRecord / s.executeTeranodeOp helpers added in the previous commit. Each call site shrinks because the aerospike.NewValue / aerospike.BoolValue wrappers are absorbed by the helper. Behavior is unchanged when aerospike_use_native_teranode_ops is false (the default) or when the cluster's capability probe rejects the new opcode — both result in the helper producing the same NewBatchUDF / client.Execute call as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The BSV fork of aerospike-client-go-v8 is now published at github.com/bsv-blockchain/aerospike-client-go/v8 v8.7.1-bsv1 — adds TeranodeModifyOp / TeranodeReadOp wire opcodes 200/201 alongside the upstream API. Mechanical sed across 55 files renaming all imports from github.com/aerospike/aerospike-client-go/v8 to github.com/bsv-blockchain/aerospike-client-go/v8. Build, vet, and the package test suite all green. Drops the local-filesystem replace directive — the require directive now resolves directly via the Go module proxy. The previously-pinned upstream github.com/aerospike/aerospike-client-go/v8 v8.4.2 remains as an indirect dependency of unrelated transitive imports. The fork is functionally a strict superset of upstream: when aerospike_use_native_teranode_ops is false (default) and/or the cluster's capability probe rejects the new opcode, Teranode's calls go through unchanged NewBatchUDF / client.Execute paths, behaving identically to upstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picks up 15d73f5 on the BSV go-client fork which fixes routing classification for TeranodeReadOp. The original v8.7.1-bsv1 had TeranodeReadOp falling into the operate_args.go default branch and being routed via PartitionForWrite because the client uses explicit op-type switches that didn't include _TERANODE_READ. Caught on PR review of bsv-blockchain/aerospike-client-go#1. No source changes here — only the version pin moves and go.sum updates with the new tarball hash. Build, vet, mod-tidy all green.
|
🤖 Claude Code Review Status: Complete Current Review: No new blocking issues found. The native operate-path feature ( A few things worth keeping in mind (no action required, all documented):
History:
|
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “native operate-path” for mod-teranode operations in the Aerospike UTXO store (via TeranodeModifyOp, wire op 200) with a startup capability probe and transparent fallback to the legacy Lua UDF path. The PR also migrates imports to the github.com/bsv-blockchain/aerospike-client-go/v8 fork and hardens/extends response parsing + diagnostics to accommodate native-path return types.
Changes:
- Introduces native-op wrappers (
teranodeBatchRecord,executeTeranodeOp) plus capability probing and msgpack payload encoding. - Refactors multiple call sites to route through the wrappers and improves batch-result diagnostics / response parsing.
- Switches Aerospike client imports to the BSV fork and updates module dependencies/tests accordingly.
Reviewed changes
Copilot reviewed 82 out of 83 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| util/uaerospike/mock.go | Switch Aerospike client imports to BSV fork. |
| util/uaerospike/client.go | Switch Aerospike client imports to BSV fork. |
| util/uaerospike/client_test.go | Switch Aerospike client imports to BSV fork. |
| util/aerospike.go | Switch Aerospike client imports to BSV fork. |
| util/aerospike_test.go | Switch Aerospike client imports to BSV fork. |
| test/utils/containers/container_manager.go | Switch Aerospike client imports to BSV fork. |
| test/sequentialtest/large_tx_reorg/helpers.go | Switch Aerospike client imports to BSV fork. |
| test/longtest/stores/utxo/aerospike/aerospike8_test.go | Switch Aerospike client imports to BSV fork. |
| test/e2e/daemon/wip/aerospike_helpers.go | Switch Aerospike client imports to BSV fork. |
| test/e2e/daemon/ready/catchup_test.go | Switch Aerospike client imports to BSV fork. |
| test/e2e/daemon/ready/aerospike_helpers.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/verify_children_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/unmined_iterator.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/unmined_iterator_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/un_spend.go | Route single-record unspend through executeTeranodeOp and improve error context. |
| stores/utxo/aerospike/test_helper.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/teranode.go | Broaden Lua/native response parsing to tolerate native numeric/map/slice types. |
| stores/utxo/aerospike/teranode_test.go | Add tests for native-style response shapes (typed slices, int64 keys/values, string maps). |
| stores/utxo/aerospike/spend.go | Route spend-related UDF calls via wrapper; add robust per-record diagnostics and nil-safety. |
| stores/utxo/aerospike/spend_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/spend_expressions.go | Improve error propagation/diagnostics and nil-handling in expression spend path. |
| stores/utxo/aerospike/set_mined.go | Route setMined batch UDF calls via wrapper; improve per-record error/parse diagnostics. |
| stores/utxo/aerospike/set_mined_expressions.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/set_mined_expressions_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/refactored_methods_test.go | Update tests for modified setMined batch error helper signature/message. |
| stores/utxo/aerospike/pruner/pruner_service.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/pruner/pruner_service_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/pruner/mock_index_waiter_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/pruner/external_pruning_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/preserve_expressions.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/preserve_expressions_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/native_op.go | New native-op implementation: sub-op IDs, msgpack encoding, wrappers, capability probe, init hook. |
| stores/utxo/aerospike/native_op_test.go | Add tests for payload encoding shape + result-bin naming invariants. |
| stores/utxo/aerospike/mock.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/longest_chain.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/longest_chain_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/locked.go | Route setLocked via wrapper and improve batch-result validation/diagnostics. |
| stores/utxo/aerospike/index_test.go | Use shared Aerospike test-container helper (supports image overrides). |
| stores/utxo/aerospike/get.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/duplicate_spend_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/delete.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/create.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/create_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/create_external_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/container_helper_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/container_helper_internal_test.go | New helper to run Aerospike containers with env-controlled image/platform. |
| stores/utxo/aerospike/consistency_scan.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/consistency_scan_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/conflicting.go | Route setConflicting batch UDF calls via wrapper. |
| stores/utxo/aerospike/batch_diagnostics.go | New diagnostics helpers for BatchRecord/Record/bin/value shape logging. |
| stores/utxo/aerospike/batch_diagnostics_test.go | Add tests for diagnostics helper output and nil-safety. |
| stores/utxo/aerospike/alert_system2_test.go | Switch Aerospike client imports to BSV fork. |
| stores/utxo/aerospike/alert_system.go | Route freeze/unfreeze/reassign through wrapper and harden batch result parsing. |
| stores/utxo/aerospike/aerospike.go | Initialize native-op support during store construction; route preserveUntil via wrapper; improve diagnostics. |
| stores/utxo/aerospike/aerospike_test.go | Use shared Aerospike test-container helper; optionally assert native-op enablement via env. |
| stores/utxo/aerospike/aerospike_server_test.go | Switch Aerospike client imports to BSV fork. |
| settings/settings.go | Add aerospike_use_native_teranode_ops setting wiring (default false). |
| settings/aerospike_settings.go | Document and add UseNativeTeranodeOps config field. |
| services/asset/httpimpl/GetTxMetaByTXID.go | Switch Aerospike client imports to BSV fork. |
| services/asset/httpimpl/GetTxMetaByTXID_test.go | Switch Aerospike client imports to BSV fork. |
| go.mod | Add BSV Aerospike client fork dependency; add msgpack; remove local replace; update indirects. |
| go.sum | Update dependency checksums for new fork + msgpack and bumped transitive deps. |
| cmd/monitor/monitor.go | Switch Aerospike client imports to BSV fork. |
| cmd/aerospikereader/aerospike_reader.go | Switch Aerospike client imports to BSV fork. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark Comparison ReportBaseline: Current: Summary
All benchmark results (sec/op)
Threshold: >10% with p < 0.05 | Generated: 2026-06-12 14:52 UTC |
Production asset pods were OOMKilling under load on /api/v1/subtree_data, manifesting downstream as nginx "upstream prematurely closed connection while reading response header from upstream". Goroutine profiles showed 60K+ goroutines accumulating in the chunk-fetch fan-out before SIGKILL. The earlier streaming work bounded per-request memory but did nothing to cap concurrent on-demand subtreeData file creations: each one holds chunkSize tx-metadata batches in memory, multiplied by however many clients arrive at once. With large transactions and slow clients the process trivially exceeds even a 64Gi limit. Asset side - admission control: - New asset_concurrency_subtree_data_create setting (default 4) gates the dualStreamWithFileCreation path with non-blocking TryAcquire. When the cap is reached, requests get HTTP 503 with Retry-After: 1 instead of waiting up to 30s for a permit. - Restructured GetSubtreeDataReader to check Exists first without holding the reader semaphore. File-exists fast path uses the existing reader sem; on-demand creation uses the new create sem. - Defensive cap on the pending chunk map in writeTransactionsViaSubtreeStoreStreaming (2 * concurrency); aborts with a clear error if a future scheduler regression grows it. - ctx check every 256 txs in writeChunkToWriter so a client disconnect releases chunkMetaSlice promptly instead of waiting for the next pipe write to fail. HTTP utility - typed 503 + retry helper: - buildHTTPError now produces errors.ErrServiceUnavailable on 503 so callers can errors.Is on it. - New DoHTTPRequestBodyReaderWithRetry: exponential backoff (250ms -> 5s, max 6 attempts), honors Retry-After header, retries only on 503. Non-503 errors and ctx cancellation return immediately. Callers - retry on peer 503: - subtreevalidation/SubtreeValidation.go (getSubtreeMissingTxs) - subtreevalidation/check_block_subtrees.go (CheckBlockSubtrees) - blockvalidation/get_blocks.go (fetchSubtreeDataFromPeer) Tests: - 7 new unit tests for the retry helper covering success, retry-then- succeed, attempt exhaustion, Retry-After honoring, no-retry on non- 503, ctx cancellation, and parseRetryAfter parsing. Race-clean. Verified: go build, go vet, go test ./util/ -race all clean.
Picks up the latest changes on github.com/bsv-blockchain/aerospike-client-go/v8. go.sum tarball hash refreshed; no source changes in this commit. Pin: v8.7.1-bsv2.0.20260508103936-b18e5d80a9e4 (commit b18e5d8).
Adds aerospike_enable_client_metrics (default true, preserving prior behavior). When disabled, util/aerospike.go skips client.EnableMetrics(nil) and the stats polling goroutine entirely. Motivation: under sustained heavy batch load we have observed thousands of goroutines blocked on a single sync.RWMutex.Lock() inside aerospike-client-go's per-node nodeStats updateOrInsert path, called from writeCommand.parseResult on every batch result. EnableMetrics is what wires that tracking up; turning it off bypasses the contention point entirely at the cost of losing the per-cluster Prometheus stats this Go process exposes. Operational guidance: - Keep enabled in normal production where observability matters. - Flip to false on clusters where nodeStats mutex contention is observed (goroutine profile shows many waiters on the same mutex address inside aerospike-client-go internals) until a client-side fix lands upstream.
…ines Production observation on dev-scale-2: subtree-validator's "Tx Meta read from Kafka /second" Grafana panel oscillates between 0 and 2.4M tx/s with mean ~1.5M, repeatedly dropping to zero. One month ago the same metric was a steady ~900K/s with no zero-drops. Cache hit rate sits at ~50% even for new (not-yet-mined) transactions, and subtree validation takes 30-49s per subtree (target: ~2s). Root cause is in services/subtreevalidation/txmetaHandler.go: - Each Kafka message spawns a goroutine via `go func()` (unbounded fan-out, since 9f4f1e5 in Dec 2025). - Each Kafka message carries a binary BATCH of N entries (since 4dcf264 in Dec 2025), but the handler loops calling SetCacheFromBytes once per entry sequentially. - Each SetCacheFromBytes call takes a per-bucket write lock in improved_cache.bucket.Set (improved_cache.go:799). Under burst load the cumulative effect is thousands of goroutines serializing through the cache's bucket locks. The cache's own sharded-bucket parallelism (8192 buckets in SetMulti) is wasted because each goroutine takes locks one at a time. Throughput collapses to single-writer speed, recovers when contention drains, collapses again — the visible 0/2.4M oscillation pattern. The franz Kafka library switch in #611 (2026-03-27) is the most likely trigger for the recent regression, but the underlying design flaw predates that. Fix: - New txmetaCacheJob struct: parsed Kafka batch (deep-copied keys/ values, separated ADD/DELETE). - Bounded worker pool consumes parsed jobs from a buffered channel. Workers call SetCacheMulti ONCE per batch — letting the cache's bucket fan-out parallelize internally, taking each touched bucket lock once per Kafka message instead of once per entry. - Handler now does cheap parsing on the Kafka consumer goroutine, then enqueues onto the work channel. Channel-full enqueue BLOCKS, applying backpressure to Kafka rather than letting goroutines pile up unboundedly. - Shutdown is driven by closing the work channel (not ctx cancel) so workers drain queued items before exiting. sync.Once on the close keeps Stop() idempotent for tests with deferred cleanup. Settings: - subtreevalidation_txmetaCacheKafkaWorkers (default 8) - subtreevalidation_txmetaCacheKafkaQueueSize (default 256) Interface: - SetCacheMulti added to txMetaCacheOps. TxMetaCache already has it. Tests: - Existing TestServer_txmetaHandler covers nil/short/ADD/DELETE paths; updated to assert SetCacheMulti (not SetCacheFromBytes) on ADD and run against the worker pool. - New TestServer_txmetaHandler_BatchesIntoSingleSetCacheMulti guards the regression: 50 entries -> ONE SetCacheMulti call. - New TestParseTxmetaBatch covers empty/zero-entry/multi-entry/ truncated batches and verifies keys/values are deep-copied so later mutation of the source buffer doesn't corrupt parsed jobs. - Race detector clean. Note: this fix does NOT remove the cache writeback in TxMetaCache.BatchDecorate (cache-aside on store fetches) — that path is needed for nodes whose Kafka publishers don't carry every transaction. Verified: go build, go vet, go test -race for the changed packages all green; pre-commit hooks pass.
Restores the partition-level consume parallelism that the franz-go switch (#611, 2026-03-27) silently removed, and fixes the producer-side sticky-partitioner skew that produces bursty per-partition load. Three changes, all on the txmeta hot path: 1) services/validator/Validator.go — Kafka producer now sets Key to a tx hash (the first hash in the batch for batched sends, the txn's own hash for single sends). Previously Key was nil, which under franz-go's default StickyKeyPartitioner is equivalent to a StickyPartitioner: every batch lands on the same partition until the linger/batch threshold trips, producing the bursty oscillation we observed in production. With a non-nil key, StickyKeyPartitioner hashes the key onto a partition deterministically. tx hashes are uniformly distributed, so partition usage is now uniform. 2) util/kafka/kafka_consumer.go (Start) — replaces the single-goroutine fetches.EachRecord(...) loop with fetches.EachPartition(...) + one goroutine per partition per fetch. Within a partition records are still processed sequentially so per-partition order is preserved; across partitions the work runs in parallel. partitionWg is awaited before the next PollFetches so in-flight goroutine count is bounded by partition count. This restores N-way concurrency that the previous Kafka library provided implicitly via per-partition consumer goroutines. 3) util/kafka/kafka_consumer.go (NewKafkaConsumerGroup) — when AutoCommitEnabled is true, registers kgo.AutoCommitMarks() and the per-partition loop calls client.MarkCommitRecords(record) only after consumerFn returns nil. With the previous default auto-commit any record that consumerFn errored on was still being committed by the auto-commit timer because franz-go tracks "iterated" not "succeeded". This change makes auto-commit consistent with the manual-commit path's semantics. Verified: - go build / go vet across changed packages — clean - go test -short -race ./util/kafka/... — pass (perf tests are -short-skipped; they require a real broker) - go test -race ./services/subtreevalidation/{TestServer_txmetaHandler, TestParseTxmetaBatch} — pass (worker pool from #834 still works under the new parallel feeder) - Pre-commit hooks (gofmt / gci / golangci-lint) — pass Notes: - Producer compression intentionally NOT enabled — Bitcoin tx data is high-entropy, compression would just burn CPU. - AutoCommitMarks closes the gap where errored records were still committed; it does NOT close the gap where async handlers (txmeta worker pool from #834) return before the cache write completes. Per discussion this is acceptable: cache misses fall through to the UTXO store transparently.
CI on PR #828 caught three e2e tests panicking with "send on closed channel" originating from txmetaHandler.go:148: legacy-sync TestBIP68_TimeBased_Accept smoketest TestBIP68_HeightBased_Accept prunertest TestPrunerUnminedParentRetention All three use the in-memory Kafka consumer which delivers a final message AFTER Server.Stop() has returned from txmetaConsumerClient. Close(). The handler running on that delivery race-loses with stopTxmetaCacheWorkers's close(txmetaCacheJobCh) and panics. Fix: a sync.RWMutex (txmetaCacheCloseMu) plus a closed flag (txmetaCacheClosed) coordinate the two paths: - Senders (txmetaHandler) take the read lock, check closed, send. - Closer (stopTxmetaCacheWorkers) takes the write lock, flips the flag, closes the channel — all under the write lock. Read-lock-then-check-then-send is atomic from the closer's POV: a sender either sees closed=false and completes its send before the write lock is granted, or sees closed=true and bails. No path can land on close(ch) followed by send(ch). Read locks are uncontended on the steady-state hot path and the close runs once per Server lifetime, so overhead is negligible. Test: TestServer_txmetaHandler_ShutdownRace stresses 16 sender goroutines against a concurrent stop(); 10× -race iterations clean. Verified: - go build / go vet — clean - go test -count=10 -race -run "TestServer_txmetaHandler| TestParseTxmetaBatch" ./services/subtreevalidation/ — pass - Pre-commit hooks — pass
… size, clamp NaN/Inf Three correctness fixes from review of the original PR: 1. settings/settings.go was missing the imperative loader entry for aerospike_semaphore_multiplier, so the struct-tag default of "1.0" never took effect — settings.NewSettings() returned 0.0 and every deployment without an explicit override silently ran with the in-process semaphore disabled. Add the getFloat64 call so the documented 1.0 default actually applies. 2. Client.GetConnectionQueueSize returned cap(nil) = 0 when the semaphore was deliberately disabled, which made the pruner's recommendedMax = 0 × threshold and collapsed chunkGroupLimit to 1. Store the resolved underlying connection-queue size on Client and report it when the semaphore is nil so external pool-capacity heuristics survive an opt-out. 3. buildConnSemaphore did not guard NaN/+Inf or runaway scaled values before make(chan, ...). NaN now disables the semaphore (treated as garbage input); +Inf and any scaled value above maxSemaphoreCapacity (1<<20) are clamped so a typo like 1.0e10 cannot allocate a multi-GB channel. Adds a settings.NewSettings() regression test that asserts the runtime default is 1.0 — the existing tests bypass the broken seam by calling newClientConfig / buildConnSemaphore directly. Also extends TestBuildConnSemaphore with NaN/+Inf/clamp cases and adds an explicit fallback test for GetConnectionQueueSize.
buildOOBFixture and buildInRangeFixture created their SyncedMap with limit=2 and then inserted exactly 2 entries. The _NilParentTx tests subsequently Set on an existing key; once len == limit, go-tx-map's setUnlocked evicts a random item from the map before writing — which ~50% of the time drops the child the SUT is about to look up, leaving the test failing with "tx <hash> not found in txMap". Reproduced locally with -count=20 under -race. Use an unbounded SyncedMap in the fixtures (the production path sizes by block tx count, not 2) and leave a comment explaining the eviction trap so future fixtures don't fall into the same hole. Duplicates the fix in PR #931 so this PR's CI is unblocked independently; will fold into theirs if #931 lands first.
# Conflicts: # go.mod # services/legacy/netsync/handle_block_test.go # stores/txmetacache/improved_cache_test.go
# Conflicts: # go.mod # go.sum # settings/aerospike_settings.go # settings/settings.go # stores/txmetacache/cache_backend.go # stores/txmetacache/improved_cache_meta.go # stores/txmetacache/pointer_cache.go # stores/txmetacache/pointer_cache_test.go # stores/txmetacache/txmetacache.go # stores/txmetacache/txmetacache_test.go # stores/utxo/aerospike/spend_expressions.go
PR #957 landed on main using the stock aerospike-client-go/v8 import path. pr-828's whole utxo/aerospike package uses the BSV fork (github.com/bsv-blockchain/aerospike-client-go/v8), which adds the TeranodeModifyOp wire opcode that the native-ops path needs. Mixing stock and fork in the same package would break type compatibility across files. Same surgical edit as was applied to spend_expressions.go during the prior conflict resolution. The BSV fork is API-superset of stock for everything circuit_breaker.go touches, so the change is import-path-only.
# Conflicts: # go.mod # go.sum
|
# Conflicts: # stores/utxo/aerospike/locked.go # stores/utxo/aerospike/spend.go # util/aerospike.go
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
oskarszoon
left a comment
There was a problem hiding this comment.
Re-review against 2988d2df7. Prior round cleared: the three Copilot findings (WritePolicy, probe doc/impl, PreserveTransactions nil-slice) all verify as fixed, merge conflict resolved, fork re-pinned to v8.7.1-bsv3. One blocker remains.
Blocking — native unspend drops the #766 SpendingData ownership check (#899). un_spend.go:164-180 routes through executeTeranodeOp(subOpUnspend) when aerospike_use_native_teranode_ops=true. The native subOpUnspend=3 dispatcher doesn't enforce the ownership comparison the Lua path does (teranode.lua:513-540), so with the flag on an unspend can clear a spending_data slot it doesn't own → resurrects a spent UTXO and corrupts spentUtxos accounting on the reorg / ProcessConflicting path. Default-false plus a doc note isn't a sufficient guard for consensus state — one flag flip disables the check silently. Fence unspend to the UDF path until the server dispatcher lands #899 (keeps the native win for the other ops), or gate the setting on a server version that includes the fix.
Before this flag is trusted in prod (non-blocking for a default-false merge):
- Startup probe validates only
setLocked; one bool gates all sub-ops and there's no runtimePARAMETER_ERROR→UDF fallback. Probe pass ≠ proof the other sub-ops decode/enforce correctly. (native_op.go:194-291) - Native path msgpack-marshals raw Go types (
uint32vsint) untranslated, unlike the UDF path's normalised values. A numeric-type mismatch on any sub-op (e.g.spendMulti'suint32blockHeight) passes the probe and then fails/corrupts at the first real op. Normalise numeric args toint64, or accept bothMP_INT/MP_UINTserver-side, and verify every sub-op against the CSUBOP_TABLE. - Path selection + probe error-classification have zero unit coverage (
Store.clientis concrete*uaerospike.Client, no mock seam). Add an interface seam + a conformance test (spend as B, attempt unspend with A's SpendingData, assert the UTXO stays spent) alongside the server change.
ordishs
left a comment
There was a problem hiding this comment.
Review — native operate-path for mod-teranode UDFs
The headline feature (opt-in TeranodeModifyOp wire-200 path with a single decision point in teranodeBatchRecord / executeTeranodeOp, a startup capability probe biased toward false-negatives, transparent UDF fallback, shared BatchWritePolicy, and a msgpack-tolerant response parser) is well-architected and conservatively gated. Off by default → byte-identical to current behaviour. Init ordering is correct: client/namespace/setName/logger are all set before initNativeTeranodeOps runs, and the probe never touches the logger when the setting is off.
Acknowledging up front that this PR bundles several independently-shippable changes specifically because they've soaked in production for a couple of weeks. That's real evidence, and it retires my weakest argument — so the points below are deliberately split into "what soak time covers" vs "what it doesn't."
What production soak reasonably retires
The nil-hardening, diagnostics helpers (describe*, with tests), streamOrAbort, the catchup ErrServiceError→ErrExternal reclassification, and the Prometheus metric rename/removal all ship live regardless of the flag. Two weeks of exposure is worth more than fresh review eyes, so I'm downgrading "these should be separate PRs" from a blocker to a preference. One caveat that soak doesn't cover: this repo is consumed downstream, so the metric rename (set_tx_meta_cache_kafka → _kafka_batch + new _kafka_count) and the removal of tx_meta_cache.hit_old_tx will break other operators' dashboards even though they're already reconciled here. Worth a CHANGELOG/release-note callout.
What soak does NOT automatically cover — depends on flag state
The entire native path is short-circuited to the identical UDF code when the flag is off. So the key question is whether aerospike_use_native_teranode_ops was actually true in the soaked deployment:
- If it was OFF — everything except the native path has production evidence, but the headline feature has zero production exposure.
- If it was ON — the native path soaked (great), but two concerns were then live in production:
1. unspend ownership check skipped on the native path (security regression, #899). un_spend.go forwards SpendingData to subOpUnspend=3, but the server-fork dispatcher doesn't yet validate it. PR #766 made that ownership check mandatory on the UDF path; the native path weakens spend-reversal authorization. Unspend fires on reorgs / block invalidation — so even "flag on in prod" only counts as tested here if a reorg actually occurred in that window. This is the one place the PR can weaken a correctness guarantee rather than just add a code path, so it's my primary ask.
2. No runtime PARAMETER_ERROR → UDF fallback. The probe samples one partition master. If it passes but a runtime spend/setMined lands on a stock-Aerospike master (mid-rolling-upgrade or a rolled-back node), the write hard-fails on a consensus-critical path with no fallback. A uniform fork-only cluster never exercises this, so soak time doesn't retire it — the gap is still there for the next rolling upgrade.
Asks before this is enabled anywhere new (neither requires splitting the PR)
- Confirm the flag state in the soaked deployment, and whether unspend/reorg was exercised under it.
- Add one line to the setting's
longdescflagging the #899 unspend-ownership gap — right now that caveat lives only in the PR body and the issue, not where an operator toggling the flag will see it.
Smaller notes
- No CI-level test of msgpack wire-contract fidelity per sub-op (esp.
spendMulti's[]aerospike.MapValuewith non-string keys). The frozen wire contract could drift silently; a golden-bytes encode test per sub-op would guard it. Acknowledged as a follow-up. - Probe writes into the production namespace/set (unique per-process key + 60s TTL — reasoning is sound); worth confirming it can't trip secondary-index population on that set.
- The new
ErrExternalcatchup branch inblockvalidation/Server.gochanges peer-failover behaviour and has no test. - Sub-op id 12 (
setDeleteAtHeight) reserved-but-unused may trip an unused-const linter.
Verdict
Comfortable with this merging as a production-validated batch — dropping the split-it-up insistence. The two items I'd still want resolved before the flag is enabled on any new cluster are #1 and #2 above, and the soak history only addresses them if the flag was on and a reorg/unspend actually occurred. Static review only; I did not run the suite (the native path needs the BSV-forked server to exercise).
The native subOpUnspend=3 dispatcher in the BSV fork of aerospike-server now enforces the #766 SpendingData ownership check. Update the stale un_spend.go comment that said the dispatcher still needed the fix, document the minimum server-build requirement in the UseNativeTeranodeOps longdesc, and drop the private-repo reference from the native_op.go header comment.
|
Re: the native This is now resolved server-side — the native On the teranode side, no client change was required: This PR updates the now-stale On the non-blocking follow-ups (probe validates |
|


Summary
Adds an opt-in path that invokes mod-teranode UDF functions through the native operate-path (
TeranodeModifyOp, wire op type 200) instead of the legacy UDF executor. The native path:LIST_APPEND/CDT_MODIFY(finer-grained than UDF context lock)Expected to give meaningful concurrency wins under heavy batch load. The feature is off by default; existing clusters keep working unchanged.
Required server
The BSV fork of
aerospike-server-private(branchfeat/teranode-native-opor merged equivalent), which adds the matching server-side dispatcher (AS_MSG_OP_TERANODE_MODIFY= 200, sub-op id table 1..13).Clusters running stock Aerospike or older versions of the BSV fork are unaffected — the capability probe (described below) detects unsupported servers and falls back to the UDF path.
Important: mixed-version Aerospike clusters are NOT supported. The capability probe samples one partition's master only; a mid-rolling-upgrade cluster can produce false-positive or false-negative probe results. Upgrade the aerospike-server cluster fully before flipping
aerospike_use_native_teranode_ops = true. See the setting'slongdescfor details.Decision wrapping
Each call site delegates path selection to a single helper:
The helper returns either an
aerospike.NewBatchWrite(... TeranodeModifyOp(...))oraerospike.NewBatchUDF(...)BatchRecord, depending on:aerospike_use_native_teranode_ops(defaultfalse)setLockedsub-op viaTeranodeModifyOpagainst a short-lived probe key. A patched BSV server recognises wire op 200, executes the sub-op, and returns a structured response — that's a "supported" signal. A stock/unpatched server rejects the unknown opcode withPARAMETER_ERROR, which is treated as "not supported". Anything else (timeout, other error, connection drop) is also treated as "not supported", biasing toward false-negatives so the fallback never runs against a server that doesn't support the op.If either gate is closed, the helper produces the same
NewBatchUDFcall the existing code did. Behavior is byte-identical to the current code path on clusters that don't enable the setting or don't run the BSV server fork.Split into separate PRs for independent review
Three orthogonal themes that were originally bundled in this branch have been factored out so reviewers can sign off on each in isolation. Their content is still present on this branch so PR #828 remains independently buildable; once each lands on main, the next merge from main will drop the duplicate hunks naturally.
codes.Unimplemented(MERGED into main)Known follow-ups (deferred to separate PRs)
PARAMETER_ERROR→ UDF fallback interanodeBatchRecord/executeTeranodeOp. Closes the rolling-upgrade gap where the capability probe passes (one partition's master is on the BSV fork) but a runtime write hits a stock-Aerospike master. Mirrors thecodes.Unimplementedfallback in fix(blockassembly): fall back to row-oriented AddTxBatch on Unimplemented #897. Will be a separate PR.unspendSpendingData ownership check on the native path — issue #899. PR #766 added a mandatorySpendingDataarg to the UDF unspend for ownership verification; the native-opsubOpUnspend = 3dispatcher needs the same arg before the native path can safely apply it. Until the server-fork side ships the matching dispatcher change, the native path skips the check — a security regression vs the UDF path. Workaround: keepaerospike_use_native_teranode_ops = falseon any cluster where the ownership check matters.teranodeBatchRecordUDF-vs-native path routing — currently the path selection is only exercised by integration tests against a live BSV-forked cluster.72616cb3187f(e.g.v8.7.1-bsv3) and re-pin via tag. Currently pinned via Go pseudo-version, which is integrity-safe (go.sum hash) but operationally unclear.What's in the diff
Foundation (commit 1)
stores/utxo/aerospike/native_op.go(new): sub-op id constants, msgpack[sub_op_id, args]payload encoder,teranodeBatchRecord/executeTeranodeOpwrappers, and the capability probe.stores/utxo/aerospike/aerospike.go: addsStore.useNativeTeranodeOpscached field ands.initNativeTeranodeOps(ctx)invocation inNew(...).settings/aerospike_settings.go+settings/settings.go: addsAerospikeSettings.UseNativeTeranodeOps bool(default false), keyedaerospike_use_native_teranode_ops.Cutovers (commit 2)
11 mod-teranode UDF call sites swapped from
aerospike.NewBatchUDF(...)/s.client.Execute(...)tos.teranodeBatchRecord(...)/s.executeTeranodeOp(...):set_mined.goaerospike.goalert_system.go(×3)conflicting.golocked.gospend.go(×3)un_spend.goExecute)Net effect:
-3lines across the call-site files (the wrapper absorbsaerospike.NewValue(...)/aerospike.BoolValue(...)ladders, so each call site shrinks).Module path switch (commit 3)
github.com/aerospike/aerospike-client-go/v8imports across 55 files togithub.com/bsv-blockchain/aerospike-client-go/v8.replacedirective.The fork is consumed as a regular dependency at the Go pseudo-version
v8.7.1-bsv2.0.20260518145742-72616cb3187fvia the Go module proxy. This is 8 commits ahead of thev8.7.1-bsv2tag — the upstream fork has not yet tagged this commit. Integrity is preserved via thego.sumhash; operationally we should tag (e.g.v8.7.1-bsv3) before merge and re-pin to the tag.The fork is functionally a strict superset of upstream; on clusters that don't enable the new path, the binary behaves identically to one built against upstream v8.7.0.
Sub-op id assignment (frozen wire contract)
These IDs must match the SUBOP_TABLE in the BSV-forked server's
mod_teranode_native_op.c. Never renumber; never reuse a freed slot.