fix(dax): break DAX RX stream create/remove storm that drops connection quality to red (#3626)#3796
Conversation
…dr#3626) On 8000-series FLEX radios, issuing `stream create type=dax_rx dax_channel=<ch>` while the bound slice already has dax=<ch> makes the radio momentarily detach the stream and re-broadcast `slice <id> dax=0` immediately followed by `slice <id> dax=<ch>` again. SliceModel mirrors every echo into daxChannelChanged, so MainWindow::onDaxChannelChanged read the transient dax=0 as a real de-assignment, removed the DAX RX stream, then re-created it on the dax=<ch> that lands ~80 ms later — a self-sustaining create/remove storm. In the aethersdr#3626 report (FLEX-8600M) the loop ran at 12–25 Hz: 20,610 `stream create type=dax_rx` and 20,609 `stream remove` commands in a single session. The churn floods the radio's TCP command channel and resets the DAX VITA-49 sequence every cycle (13,188 "first DAX packet seq=0" re-registrations), inflating PanadapterStream packetErrorCount until the network-quality monitor drops to POOR — the "connection quality goes red after 5–10 min, DAX audio drops out" the reporter sees. Fix: defer the stream teardown and re-validate at fire time. The dax=<ch> rebroadcast lands long before the 1.5 s grace window elapses, so a slice is back on the channel and the removal is declined — the loop never starts. A genuine user de-assignment has no follow-up rebroadcast, so the channel stays unwanted and the stream is removed after the grace window. The stream stays registered across the transient, so the dax=<ch> edge's create is skipped (idempotent), breaking the create side too. Full DAX RX stream refcounting remains tracked in aethersdr#3305. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
If this is specific to 8000 radios does any of this pertain to the original #3669 bug still occurring in a 6000 radio? TNX 73 |
…generation race (aethersdr#3626) Defensive-review follow-up to the storm fix. Two hardening tweaks, no behavior change to the happy path: - scheduleDaxRxStreamRemoval: the in-flight QTimer::singleShot was not cancelled by stopDax() (singleShot has no handle), and the fire-time lambda only guarded on m_daxBridge. A DAX off→on toggle inside the 1.5 s grace window leaves m_daxBridge non-null again, so a stale timer from the previous bridge generation could fire and spuriously `stream remove` a channel the new bridge wants (mitigated by re-validation, but real). Make the pending set authoritative: bail when QSet::remove(ch) reports the key already gone — stopDax()'s clear() now genuinely cancels in-flight fires. - Document that the unconditional `slice set <id> dax=<ch>` re-assert is storm-safe only because SliceModel::applyStatus drops same-value dax echoes; that equality guard is load-bearing for this fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ten9876
left a comment
There was a problem hiding this comment.
Excellent root-cause from the 847k-line log. Verified the storm-break logic on both sides:
- Remove side: scheduleDaxRxStreamRemoval() defers teardown 1.5s and re-validates at fire time (if any slice still has daxChannel()==ch, decline). The dax=ch rebroadcast lands ~80ms later (≪ grace), so the removal is declined and the loop never starts; a genuine de-assignment has no rebroadcast so it's removed after grace.
- Create side: broken automatically — the stream stays registered, so the daxStreamIdForChannel(newCh)==0 guard (MainWindow_DigitalModes.cpp:1125) skips the create.
- Timer lifecycle clean: singleShot (no leak), m_daxPendingRxRemoval dedups so timers don't stack, stopDax() clears the set so a stale fire from a previous bridge generation self-cancels (if (!remove(ch)) return), with re-validation as defense-in-depth. TCI/RADE ownership guards preserved.
- Good call documenting that SliceModel's same-value-echo drop (if (m_daxChannel != ch)) is now load-bearing for this fix.
This also removes another packetErrorCount source feeding the network-quality monitor (same chain as #3810).
Caveat acknowledged: not live-validated on the failing path (DAX HAL plugin absent on the test Mac) — proven by code-trace + the reporter's log. Merging given the contained, airtight change and that 8000-series users are hitting red connections; the reporter should confirm on a DAX-HAL machine. Thanks @jensenpat.
Hardware validation — FLEX-8400M, both DAX paths ✅Post-merge validation on a FLEX-8400M (8000-series, fw 4.2.18). Deployed the build to a test Mac and exercised both the TCI and DAX-audio-bridge paths, then pulled logs. Radio-side trigger confirmed on hardware. On every TCI path ( DAX bridge path (
200 triggers → 8 creates, ~195 deferred teardowns declined by re-validation, connection healthy throughout. A pre-fix build would have been visibly ramping inside this window. Fix confirmed working on real hardware. The stale-timer bridge-generation hardening ( Out of scope / follow-up: every |
Fixes #3626
Symptom
On a FLEX-8600M (macOS), connection quality reads EXCELLENT for ~5–10 min
(sometimes ~30 s), then drops to POOR / red with the packet-loss graph
ramping up. While degraded, DAX recordings crackle and chop badly. A restart
resets the cycle. Multiple 8000-series users report the same.
Root cause — a DAX RX stream create/remove storm
Traced entirely from the reporter's attached log (847k lines):
stream create type=dax_rxand 20,609stream removecommands in one session — a tight loop running at 12–25 Hz.seq=0" resets in the log).
The loop is a feedback cycle specific to 8000-series radios:
MainWindow::onDaxChannelChanged()issuesstream create type=dax_rx dax_channel=1for a slice that already hasdax=1.slice 0 dax=0immediately followed byslice 0 dax=1(confirmed in the log; sametransient-rebroadcast behavior noted for TCI audio (DAX 1) does not start on connect; requires mode change + DAX toggle to recover #3669).
SliceModel::applyStatusmirrors every echo intodaxChannelChanged(
SliceModel.cpp:846), so the reconciler reads the transientdax=0as areal de-assignment →
stream remove, then reads thedax=1that lands~80 ms later as a new assignment →
stream create+slice set→ backto step 2.
Why that turns the connection red
The storm floods the radio's TCP command channel and resets the DAX VITA-49
sequence on every recreate. Each reset is a sequence discontinuity counted in
PanadapterStream::packetErrorCount(), which feedsRadioModel::networkQualityTargetScore()packet-loss term → the score falls →NetStatewalks down to POOR. That is exactly the rising packet-loss graphand red status the reporter screenshotted. (The variable 5–10 min / 30 s onset
matches a user action — band/profile/slice reconfiguration — kicking off the
first transient; in this capture the storm began at a panadapter
add/remove reconfiguration ~1h27m in.)
Fix
Defer the DAX RX stream teardown in
onDaxChannelChangedand re-validate atfire time (
scheduleDaxRxStreamRemoval):dax=0schedules a removal 1.5 s out instead of firing inline.dax=1rebroadcast lands ~80 ms later, so when the timer fires a slice isback on the channel and the removal is declined — the loop never starts.
dax=1edge'sstream createis skipped (already idempotent viadaxStreamIdForChannel),so the create side is broken too.
unwanted → the stream is removed after the grace window. Existing TCI/RADE
ownership guards are preserved.
Minimal, contained change (2 files). Full DAX RX stream refcounting remains
tracked in #3305.
Proof
Broken baseline (reporter's log, build 26.6.3, FLEX-8600M): 20,610
stream create type=dax_rx+ 20,609stream remove, ramping 12 Hz → 25 Hz; 13,188 DAXstream re-registrations; trim/packet-loss climbing into the red.
Code trace: the deferred + re-validated teardown collapses each transient
dax=0→dax=1rebroadcast to a single harmlessslice setno-op, sending zerostream create/stream remove— the storm cannot sustain.Agent automation bridge — could not drive the live path (gap, see below):
the storm only runs when the DAX bridge is active (
m_daxBridge != null), whichrequires the macOS DAX HAL plugin. The plugin is not installed on the test
Mac (
startDax()logsDAX HAL plugin not installedand bails), soonDaxChannelChangednever runs. The fixed build was confirmed to build, launch,and run RX-clean against a FLEX-8400M (same 8000-series family/firmware
4.2.18) via the bridge; the DAX-bridge code path itself could not be exercised
here. Recommend validating on a machine with the DAX HAL plugin installed (or
by the reporter).
Out of scope (separate bug, filed separately)
The same logs show DAX RX audio delivered at ~2× real-time from session
start (reduced-BW packets
pcc=0x0123, 128 samples/pkt × ~375 pkt/s = 48 kHzmono), but
VirtualAudioBridge/HAL assume 24 kHz, so ~50 % is trimmed everysecond → the constant chop and "very quiet at 100 %" the reporter also reports.
That is a rate-assumption bug independent of this storm and needs its own change.
💻 Generated with Claude Code (Opus 4.8) with architecture by @jensenpat