Summary

Two complementary fixes for the txmeta producer-side latency regression that landed when the kafka client was switched from Sarama to franz-go (#611). Together they take p99 publish→consume latency from ~7 s to ~22 ms on the regression test. Already deployed to dev-scale-1/2 and confirmed working at production load — see "Post-deploy validation" below.

Commit 1: cut flush_frequency to 10ms on high-fanout topics

The Sarama → franz-go switch silently re-wired the URL query parameter flush_frequency from "max time between flushes" to franz-go's per-partition kgo.ProducerLinger. On the high-fanout txmeta topic (256 partitions, ~5 batched msgs/s/partition at peak), each partition rarely fills 1 MiB before the 1 s linger, so every record paid up to ~1 s of producer-side delay. The subtree-validator's local cache lagged the validator by 1–2 s and every subtree triggered ProcessTxMetaUsingCache → ThresholdExceededError → 1 s RetrySleep → retry.

Drops flush_frequency from 1s to 10ms on the three high-fanout topics: kafka_txmetaConfig, kafka_validatortxsConfig.operator, kafka_legacyInvConfig. Low-volume topics (invalidBlocks, rejectedTx, unitTest) keep their existing 1s.

Commit 2: decouple outer batcher linger from flush_frequency

A subtler footgun in the same area: KafkaProducerConfig.FlushFrequency was driving two lingers at once — franz-go's per-partition ProducerLinger (the user-facing knob), and the outer async-batcher's straggler-flush timer (an internal implementation detail). Setting flush_frequency=1s stacked two 1-second lingers on the same publish path.

Introduces a new URL query param outer_batcher_linger (field: OuterBatcherLinger, default 10ms) controlling only the outer batcher. flush_frequency now controls only kgo.ProducerLinger, which is what an operator looking at the URL expects.

Pre-deploy evidence

Production (Prometheus, dev-scale-1/2, Friday May 8 2026, 18:00–21:00 UTC at 1.28 M TPS peak):

• Only txmeta-dev-scale-1-scale-1 has producer-buffer backlog (mean ≈ 72 k msgs across 20 propagation pods, peak 186 k). Every other topic stays at 0.
• teranode_kafka_producer_produce_request_latency_seconds p99 ≈ 642 ms, p50 ≈ 87 ms.
• validate_subtree_retry rate ≈ 1–2 / s, matching the ~1.2 subtree/s rate — basically every subtree retries.
• validate_subtree_duration p99 mean = 16 s, max 127 s.
• bless_missing_transaction_count rate = 0 — retries always eventually succeed; the cache does fill, it just lags.

TestLingerLatencyRegression (OrbStack-backed Redpanda, 32-partition topic, 200 records 25 ms apart):

Post-deploy validation

The matching configmap patch (flush_frequency=1s → flush_frequency=5ms on txmeta and legacyInv) was applied to dev-scale-1/2 at 2026-05-11 11:27 UTC. After ~22 min of sustained ~1.30 M TPS:

The "Tx Meta read from Kafka /second" Grafana panel is now flat at ~1.3 M/s on both pods — no near-zero dips, no scrape-induced "gaps". That panel's behaviour was the originating symptom.

One thing flagged for monitoring, not a regression: bless_missing_transaction_count is now firing at very low rates (mean 0.23/s on scale-1, 0.85/s on scale-2 with one 20.76/s burst) where it was zero before. Pre-fix that path never fired because the ThresholdExceededError → 1 s retry short-circuited every cache miss. Post-fix, the retry doesn't trigger, so genuine cache misses fall through to the legitimate "fetch from UTXO store" path. The miss rate is microscopic (≈0.00007% of txs), so this is fine — but if it grows it's the right alarm signal to surface, because it'll mean the cache is undersized rather than being masked by the retry loop.

Follow-ups (intentionally out of scope here)

1. dev-scale-1/2 configmap update. ✅ Already applied (with flush_frequency=5ms rather than the 10ms in this PR's defaults — both work). scale-1-shared-config.kafka_txmetaConfig in teranode-argocd-deployments was patched; the matching PR there should be linked.
2. 256 partitions for one Redpanda broker is over-provisioned for the actual record rate; consider dropping to 32–64. Not required for this fix, but a contributing factor to the slow broker write p99 (now moot under low-linger config).

Test plan

• go vet ./util/kafka/ clean.
• go build ./util/kafka/ clean.
• go test -short -count=1 ./util/kafka/ passes (all unit tests, including new TestNewKafkaAsyncProducerFromURLOuterBatcherLinger cases).
• go test -v -run TestLingerLatencyRegression -timeout 5m ./util/kafka/ passes locally with the numbers above.
• Deployed to dev-scale-1/2; metrics confirm the regression is resolved (see "Post-deploy validation").
• Reviewer to confirm no settings_local.conf override for terabuild / mainnet / testnet / teratestnet relies on the old semantic.

🤖 Generated with Claude Code