Skip to content

fix(dax): centralize DAX RX stream ownership in PanadapterStream (#3305) — kills the #4009 re-assert storm class#4017

Merged
ten9876 merged 7 commits into
aethersdr:mainfrom
jensenpat:fix/3305-dax-stream-refcount
Jul 5, 2026
Merged

fix(dax): centralize DAX RX stream ownership in PanadapterStream (#3305) — kills the #4009 re-assert storm class#4017
ten9876 merged 7 commits into
aethersdr:mainfrom
jensenpat:fix/3305-dax-stream-refcount

Conversation

@jensenpat

@jensenpat jensenpat commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements #3305: replaces the three hand-rolled, mutually-peeking DAX RX ownership trackers (DAX bridge, TCI, RADE) with refcounted acquireDaxChannel(ch, consumer) / releaseDaxChannel(ch, consumer) in PanadapterStream — the single decider of when a radio-side dax_rx stream exists. RadioModel becomes the command plane (stream create/stream remove + a one-shot #1439 dax_clients nudge tied to the create) and the single, client_handle-filtered stream-status registrar.

Closes #3305. Fixes #4009. Fixes #3669. Supersedes #4010.

Related, intentionally left open: #3715 (slice-lifecycle single-authority RFC — the slice-side sibling of this stream-side change) and #3366 (previously diagnosed as a WSJT-X-side watchdog/format contract issue, not stream ownership).

Root cause this eliminates

The radio answers a re-assert of a live dax binding with a transient unbind/rebind dax=0/dax=<ch> status pair. The old code sent unconditional slice set <n> dax=<ch> re-asserts from status-echo edges (onDaxChannelChanged, ensureDaxForTci) — turning that transient pair into a self-sustaining oscillator: the ~12–15 Hz slice set dax storm of #4009, and the create/remove churn of #3626 before it. The invariant this PR enforces: no commands are ever emitted from status-echo edges. Acquire/release are idempotent, and the manager's removal grace window (1.5 s, cancelled by re-acquire) absorbs the transient pairs, so the loop is impossible by construction — not patched around.

Why not #4010's tracker patches: they break the loop but leave the echo-triggered re-assert in place and regress two paths — DAX off→on retoggle goes silent (stale tracker early-returns after the stream was removed), and slices arriving with DAX pre-assigned (profile restore / reconnect replay) never get a stream (the seeded tracker defeats the sliceAdded 0→ch transition). Details in the #4010 review discussion.

Design

Event Manager behavior
First holder acquires a channel daxStreamCreateNeeded(ch) → RadioModel sends stream create type=dax_rx dax_channel=<ch> + one-shot slice set <id> dax=<ch> (#1439, command-initiated, cannot oscillate)
Last holder releases Deferred removal after 1.5 s grace (subsumes #3626's defer+revalidate); re-acquire cancels
Radio destroys a held stream (profile load / slice teardown) Auto re-create after 500 ms backoff (#3476's re-arm, now universal — rearmDaxForProfileLoad no longer hand-removes streams)
TCP disconnect Table reset, no commands — the radio reaps all of a client's streams itself; consumers re-acquire on their reconnect re-arm paths
Slice removed with DAX active Bridge releases the channel (previously leaked a silent radio-side stream until stopDax)

Deleted: the unconditional re-assert in onDaxChannelChanged (the #4009 gain element), scheduleDaxRxStreamRemoval, every tciUsing/radeUsing cross-consumer peek (stopDax, deactivateRADE, the profile recovery pass), TciServer::ownsDaxChannel + m_tciDaxStreamIds + m_tciDaxBorrowedChannels, m_radeDaxStreamId, and RADE's private stream-status hook — which had no client_handle filter and could adopt a foreign client's broadcast stream status, silently shadowing the channel. Ownership coordination goes from O(consumers²) peeking to one enum value per consumer.

New automation bridge verbs (second commit)

This PR also adds the test surface that proves it, so the storm/co-hold/grace assertions are reproducible by any agent without log-grepping:

  • get dax — snapshot of the ownership table: per-channel holders (bridge/tci/rade), streamId, createPending, plus each slice's dax= assignment.
  • tci start|status|stop [abrupt] — in-process TCI client simulator speaking the WSJT-X dialect (init burst → ready;audio_samplerate + audio_start, binary-frame counting). stop abrupt reproduces the WSJT-X watchdog-reconnect shape for teardown/reap tests.

Documented in docs/automation-bridge.md; tools/automation_probe.py gained the tci command.

Live proof (FLEX-8400M, fw 4.2.18, as a second MultiFlex station)

Scenario Result
TCI audio_start (#4009 trigger) Exactly 1 stream create + 1 one-shot re-assert; 0 storm commands in the window (pre-fix: 12–15/s)
DAX bridge enabled during a live TCI stream DAX ch 1 acquired by bridge holders=0x3 — co-hold, no duplicate create, no re-assert
DAX bridge disabled during a live TCI stream released by bridge holders=0x2no stream remove, TCI audio uninterrupted (2106 frames, steady ~47/s) — the #3363/#2886 failure class
TCI client killed (last holder) grace → exactly 1 stream remove
Verb-driven re-run (tci startget daxtci stop abruptget dax) holders:["tci"] with live stream id → after abrupt stop: debounce → release → grace → single stream remove → channel table drains to []
App exit Radio session fully released; discovery clean

Log excerpt:

19:32:52 INF PanadapterStream: DAX ch 1 acquired by bridge holders=0x3
19:33:00 INF PanadapterStream: DAX ch 1 released by bridge holders=0x2   ← no stream remove
19:33:26 INF PanadapterStream: DAX ch 1 released by tci holders=0x0
19:33:28 INF PanadapterStream: DAX ch 1 unheld past grace — removing stream 4000008 (#3305)

Note: code comments cite docs/architecture/flex-protocol/state-machines.md (the wire-level contract, including captures of the transient unbind/rebind pair); that guide lands in a separate docs PR.

Constitution principles honored

Principle II — The Radio Is Authoritative On Live State. Status echoes are treated purely as state carriers; the client never re-asserts state in response to them.

Test plan

Checklist

  • Commits are signed (SSH)
  • No new flat-key AppSettings entries (Principle V)
  • Clean-room implementation — no proprietary source referenced (Principle IV)
  • No MeterSmoother changes
  • No transmit path modified (Principle VI) — dax_tx handling untouched
  • Automation docs updated for the new verbs (docs/automation-bridge.md)
  • No GHSA advisory needed

💻 Generated with Claude Code (claude-fable-5 7/1/26) with architecture by @jensenpat

🤖 Generated with Claude Code

@jensenpat jensenpat marked this pull request as ready for review July 4, 2026 02:57
@jensenpat jensenpat requested review from a team as code owners July 4, 2026 02:57
@jensenpat jensenpat added priority: high High priority priority: medium Medium priority and removed priority: high High priority labels Jul 4, 2026

@aethersdr-agent aethersdr-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff against the checked-out branch, with focus on the refcounting core, thread-safety, and the teardown/recreate paths. This is a strong, well-reasoned change — the "no commands from status-echo edges" invariant is the right structural fix for the #4009 storm class, and centralizing ownership into PanadapterStream collapses the old O(consumers²) cross-peeking into one clean refcount. All six CI checks are green (build, check-windows, check-macos, CodeQL, analyze cpp, a11y). Thanks for the thoroughness, @jensenpat — the get dax / tci bridge verbs making the assertions reproducible is a nice touch.

What I verified holds up

  • No emit-under-lock. acquireDaxChannel captures needCreate/streamId inside the locked block and emits daxStreamCreateNeeded only after the QMutexLocker scope closes; scheduleDaxRemovalLocked/scheduleDaxRecreateLocked only arm a timer while locked. Good.
  • No double daxStreamUnregistered. In the grace-removal lambda the entry is erased before calling unregisterDaxStream(removeId), so the inner scan finds nothing (channel stays 0, no emit) and the lambda emits once explicitly; the subsequent radio "removed" echo then no-ops (id already gone from m_daxStreamIds). Single emit per teardown — confirmed for both the radio-initiated and grace-window paths.
  • generation invalidation correctly expires in-flight deferred lambdas across re-acquire / disconnect-reset, and createPending guards against duplicate stream create.
  • registerDaxStream unlocking before emit daxStreamRegistered is correct (it's a QMutexLocker, .unlock() is valid).
  • Cross-thread signal flow is sound: PanadapterStream lives on m_networkThread, and the create/remove signals queue to the RadioModel main-thread lambdas.

Minor, non-blocking

  1. Doc/default port mismatch. docs/automation-bridge.md shows "port":40001 in the tci start example, but the code defaults to AppSettings.value("TciPort", "50001"). Worth aligning the example (or noting it's illustrative) so a reader doesn't assume 40001 is the default.

  2. TCI sim leaves m_tciSim armed on a server-initiated close. The disconnected handler only sets m_tciSimCloseReason; it doesn't null/deleteLater m_tciSim. So after the server drops the socket, tci start returns "already running" until an explicit tci stop. Fine for a test harness, but a one-line auto-clear (or a doc note) would make the reconnect-shape tests less surprising to drive.

  3. Observation, not a regression: if a stream create is emitted but the radio never answers (no stream status ever arrives), the channel stays createPending=true / streamId=0 and held, with no retry — the scheduleDaxRecreateLocked path only re-arms on radio-initiated teardown of an existing stream. This matches the old behavior (which also depended on the radio honoring the create), so it's not something this PR needs to solve — just flagging the one gap the recovery logic intentionally doesn't cover.

Nothing here blocks merge. Nice work.


🤖 aethersdr-agent · cost: $9.2020 · model: claude-opus-4-8

@ten9876 ten9876 self-assigned this Jul 4, 2026

@ten9876 ten9876 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: request changes

Deep pass on the DAX-ownership refactor (multi-agent finders + adversarial verify). The direction is right — one refcounted acquire/release table replacing three mutually-peeking trackers, and the "no commands from status-echo edges" invariant genuinely kills the #4009 storm by construction. But the new command plane is fire-and-forget with no failure handling, which reintroduces the exact wedge class (#3669) the PR claims to eliminate, plus a confirmed RADE channel-tracking bug. Each inline comment carries a fix (apply-able suggestion where clean, a code sketch where it spans files).

Blocking:

  1. stream create failure permanently wedges the channel (RadioModel.cpp:407) — fire-and-forget + !isConnected() drop, and createPending is never cleared on failure, so the channel is silently dead with no recovery until a full disconnect. This is the #3669 wedge, reintroduced for the first-create path.

Should fix in this pass (small, localized):
2. RADE's hold doesn't follow a mid-session dax= change (MainWindow_DigitalModes.cpp:1062) → RADE silent + old-channel stream leaks; the reconciler is Bridge-only.
3. Create→remove churn when the last holder releases while a create is in flight (PanadapterStream.cpp:1550) — contradicts the anti-storm goal.
5. TCI simulator is left a zombie after a server-side close (AutomationServer.cpp:4250), blocking tci start — and this is the harness the PR ships to prove the behavior.

Notes:
4. TCI orphans its stream on slice removal (Bridge-only sliceRemoved release) — pre-existing, but this PR is the place to close it.
6. generation bump missing in the recreate path. 7. daxStreamRegistered is a dead signal with a misleading comment.

Threading is clean, for the record — m_streamMutex guards all shared state, emits are outside the lock, snapshot is locked. And one finder claim was refuted: the reconnect "phantom-teardown" can't happen (radio reaps all streams on disconnect + the client_handle filter blocks stale status).

The core refcount design is sound; these are gaps in the failure/edge handling around it. Independent of the other open PRs, so nothing else is blocked.

Comment thread src/models/RadioModel.cpp Outdated
Comment thread src/gui/MainWindow_DigitalModes.cpp
Comment thread src/core/PanadapterStream.cpp Outdated
Comment thread src/gui/MainWindow_DigitalModes.cpp
Comment thread src/core/AutomationServer.cpp
Comment thread src/core/PanadapterStream.cpp
Comment thread src/core/PanadapterStream.cpp Outdated

@NF0T NF0T left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Human-directed independent review (Claude / @NF0T)

Full diff verified against jensenpat/fix/3305-dax-stream-refcount (d9ff740). Independent pass on all six items from @ten9876's review, plus the bot findings.

Confirmed blocking

Issue 1 — stream create fire-and-forget wedges channel permanently (RadioModel.cpp:407-408)

Independently confirmed. sendCommand() has no response callback. The !isConnected() early-return silently drops the signal with no feedback to PanadapterStream; acquireDaxChannel has already latched createPending=true, and the re-acquire guard (streamId==0 && !createPending) blocks all retries. The only clearer is registerDaxStream() on success — never called on a radio error reply. A slots-exhausted or connect-gap drop leaves the channel terminally wedged until full disconnect. This is the #3669 wedge reintroduced on the first-create path.

Confirmed should-fix

Issue 2 — RADE reconciler gated on m_daxBridge (MainWindow_DigitalModes.cpp)

Confirmed. The reconciler early-returns on !m_daxBridge, leaving RADE's DaxConsumer::Rade hold unreconciled when the slice's dax= channel changes mid-session: old channel stream leaks, new channel never created, RADE goes silent without any bridge activity.

Issue 3 — Create→remove churn when last holder releases during in-flight create (PanadapterStream.cpp:~1550)

Confirmed. releaseDaxChannel bumps generation (correctly expiring the grace timer), but the stream create is already on the wire. registerDaxStream re-inserts with holders==0 and schedules another removal — a full create+remove cycle for a channel nobody wants.

Issue 5 — TCI sim zombie after server-side close (AutomationServer.cpp:4253)

Confirmed. The disconnected handler only records the close reason; m_tciSim is never nulled. tci status reports running=true, and tci start rejects with "already running." This stalls the automated re-arm sequences that the PR uses as live proof — blocking the harness that proves the fix.

Confirmed notes

Issue 6 — Missing generation bump in scheduleDaxRecreateLocked

Confirmed (line ~1650: createPending=true without ++m_daxGenCounter). Every other createPending=true mutation pairs with a gen bump; this path inconsistently doesn't, making the timer-invalidation reasoning for the recreate path unsound relative to the immediately preceding reset.

Issue 7 — daxStreamRegistered dead signal

Confirmed. Zero connections in the PR branch; only daxStreamUnregistered is wired. The header comment promising "TCI invalidates its channel→trx cache on daxStreamRegistered" is unimplemented.

Issue 4 — TCI sliceRemoved orphan (pre-existing, natural place to close)

Confirmed pre-existing. TciServer has no sliceRemoved connection; slice removal doesn't zero daxChannel, so the Tci holder bit is never released on slice teardown.

Refutation corroboration

@ten9876's refuted phantom-teardown finding: independently confirmed his refutation is correct. The client_handle filter in registerDaxStream blocks foreign stream status adoption, and the radio reaps all streams on disconnect. Phantom-teardown cannot occur.

What's sound

Threading is clean: m_streamMutex guards all shared state, emits happen after lock release, the needCreate flag pattern (capture inside lock, signal outside) is correct throughout. Echo-triggered command path confirmed eliminated — onDaxChannelChanged records state only, no commands. Idempotency guards in acquire and release confirmed. Generation/lambda expiration correct except the single missing bump noted above. The refcount core design is solid; the issues are in the failure and edge paths around it.


Human-directed independent review (Claude / @NF0T)

jensenpat added a commit to jensenpat/AetherSDR that referenced this pull request Jul 4, 2026
… RADE/TCI hold reconciliation, sim teardown

Review items from @ten9876 (changes requested) and @NF0T's independent
confirmation pass:

1. (blocking) stream-create failure no longer wedges the channel:
   RadioModel sends the create via sendCmd with a response callback and
   reports nonzero replies (and connect-gap drops) to the new
   PanadapterStream::notifyDaxCreateFailed(), which clears the
   createPending latch and — while the channel stays held — retries on a
   2 s cadence. A slots-exhausted radio costs one command per 2 s and
   heals the moment a slot frees; the aethersdr#3669 wedge class stays dead on
   the first-create path too.
2. RADE's hold now follows mid-session dax= changes on its slice via a
   dedicated daxChannelChanged reconciler (independent of the
   Bridge-only reconciler, present on Windows builds too); disconnected
   at deactivateRADE.
3. Release-during-inflight-create no longer bounces create→remove churn:
   the last-release path keeps the entry when a create is in flight, so
   registerDaxStream lands on it and schedules ONE deterministic
   removal.
4. TCI now releases its channel hold when the backing slice is removed
   (pre-existing orphan, closed here as the review suggested): a
   sliceRemoved hook releases any Tci-held channel no remaining slice
   carries and invalidates the trx routing cache.
5. tci sim tears itself down on a server-side close (running=false,
   restart accepted) instead of zombie-blocking ; guarded
   against double-teardown with the stop path.
6. scheduleDaxRecreateLocked keeps the mutation⇒generation-bump
   invariant.
7. Dropped the dead daxStreamRegistered signal and fixed the header
   comment (TCI wires daxStreamUnregistered only).

Also aligned the docs/automation-bridge.md tci example with the real
default port (TciPort setting, 50001).

Verified: clean build; ctest 53/54 (theme_manager_test pre-existing/
environmental); sim-zombie fix proven live radio-free (server-side close
→ running=false + closeReason, immediate restart accepted). Radio was
In_Use during this pass, so the storm/co-hold scenarios were not re-run;
they are unchanged paths plus the guarded edge fixes above.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jensenpat added a commit to jensenpat/AetherSDR that referenced this pull request Jul 4, 2026
… RADE/TCI hold reconciliation, sim teardown

Review items from @ten9876 (changes requested) and @NF0T's independent
confirmation pass:

1. (blocking) stream-create failure no longer wedges the channel:
   RadioModel sends the create via sendCmd with a response callback and
   reports nonzero replies (and connect-gap drops) to the new
   PanadapterStream::notifyDaxCreateFailed(), which clears the
   createPending latch and — while the channel stays held — retries on a
   2 s cadence. A slots-exhausted radio costs one command per 2 s and
   heals the moment a slot frees; the aethersdr#3669 wedge class stays dead on
   the first-create path too.
2. RADE's hold now follows mid-session dax= changes on its slice via a
   dedicated daxChannelChanged reconciler (independent of the
   Bridge-only reconciler, present on Windows builds too); disconnected
   at deactivateRADE.
3. Release-during-inflight-create no longer bounces create->remove
   churn: the last-release path keeps the entry when a create is in
   flight, so registerDaxStream lands on it and schedules ONE
   deterministic removal.
4. TCI now releases its channel hold when the backing slice is removed
   (pre-existing orphan, closed here as the review suggested): a
   sliceRemoved hook releases any Tci-held channel no remaining slice
   carries and invalidates the trx routing cache.
5. `tci` sim tears itself down on a server-side close (running=false,
   restart accepted) instead of zombie-blocking `tci start`; guarded
   against double-teardown with the stop path.
6. scheduleDaxRecreateLocked keeps the mutation=>generation-bump
   invariant.
7. Dropped the dead daxStreamRegistered signal and fixed the header
   comment (TCI wires daxStreamUnregistered only).

Also aligned the docs/automation-bridge.md tci example with the real
default port (TciPort setting, 50001).

Verified: clean build; ctest 53/54 (theme_manager_test pre-existing/
environmental); sim-zombie fix proven live radio-free (server-side close
-> running=false + closeReason, immediate restart accepted). Radio was
In_Use during this pass, so the storm/co-hold scenarios were not re-run;
they are unchanged paths plus the guarded edge fixes above.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jensenpat jensenpat force-pushed the fix/3305-dax-stream-refcount branch from e566efc to 4d9c3cb Compare July 4, 2026 20:10
@jensenpat

Copy link
Copy Markdown
Collaborator Author

Review feedback addressed in 4d9c3cb6 — thank you @ten9876 and @NF0T for the depth here, and @aethersdr-agent for the verification pass. All seven items plus the doc nit, in review order:

1. (blocking) Create-failure wedge — fixed as sketched. The create now goes out via sendCmd with a response callback; a nonzero reply (slots exhausted, busy) or a connect-gap drop calls the new PanadapterStream::notifyDaxCreateFailed(ch), which clears the createPending latch and — while the channel stays held — arms a generation-guarded retry every 2 s. A persistently-failing condition costs one command per 2 s and self-heals the moment it clears (including the create-during-handshake case, which retries until isConnected()). Your suggested onConnected reconcile turned out unnecessary: the disconnected-drop path now feeds the same retry loop, and a real disconnect resets the whole table anyway.

2. RADE mid-session dax= change — fixed. activateRADE now wires a dedicated daxChannelChanged reconciler on the RADE slice (release old hold → acquire new), disconnected at deactivateRADE. It's fully independent of the Bridge-only reconciler, so RADE-without-bridge (all Windows builds) reconciles too.

3. Release-during-inflight-create churn — fixed via your option (b). The last-release path now keeps the entry when streamId==0 && createPending; when the stream status lands, registerDaxStream finds it, sees holders==0, and schedules one deterministic removal. No erase→re-insert bounce, no second create window.

4. TCI sliceRemoved orphan — closed here as suggested. TciServer now releases any Tci-held channel that no remaining slice carries (and drops the trx-cache entry) on sliceRemoved; the existing sliceAdded re-arm re-acquires when a replacement slice appears. RADE's variant is covered by item 2's reconciler plus deactivateRADE; a RADE slice removed out from under an active session doesn't currently deactivate RADE, which is a pre-existing RADE-lifecycle gap outside this PR's scope.

5. TCI sim zombie — fixed with your suggestion, plus a sender-guard so the graceful/abrupt stop path (which nulls m_tciSim before its close lands) can't double-teardown. Live-verified radio-free: server-side close → tci status reports running=false with closeReason:"server closed", and an immediate tci start is accepted.

6. Generation bump in scheduleDaxRecreateLocked — applied verbatim. The mutation⇒generation-bump invariant now holds everywhere.

7. daxStreamRegistered — dropped (signal + emit) and the header comment corrected: TCI wires daxStreamUnregistered only. Better no contract than an untested one.

@aethersdr-agent's doc nit: the tci start example now shows the real default (TciPort setting, 50001) instead of my dev box's 40001.

Verification note: clean build, ctest 53/54 (theme_manager_test pre-existing/environmental). The radio was In_Use by two stations during this pass, so the storm/co-hold live scenarios were not re-run — those paths are unchanged by this commit; the new code is the failure/edge handling above, of which the sim teardown was proven live and the rest is covered by the reviewers' own analysis plus build/tests. Happy to re-run the full tci/get dax live matrix once the radio frees up if you want it on record before merge.

💻 Generated with Claude Code (claude-fable-5 7/1/26) with architecture by @jensenpat

@jensenpat jensenpat requested a review from ten9876 July 4, 2026 20:11

@aethersdr-agent aethersdr-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @jensenpat — this is a genuinely well-executed refactor. Collapsing three mutually-peeking ownership trackers into one refcounted authority in PanadapterStream is the right shape, and the "never emit commands from status-echo edges" invariant is stated clearly and actually held throughout the diff. The header docs and the get dax / tci automation verbs make the whole thing testable without log-grepping, which is a real win.

I verified the two earlier reviewer concerns against the current head (4d9c3cb) — both are already fixed in this revision:

  1. Fire-and-forget stream create wedge (@ten9876's headline). Resolved. RadioModel::RadioModel now sends the create via sendCmd(...) with a response callback that calls m_panStream->notifyDaxCreateFailed(ch) on a non-zero code, and also calls it immediately on the !isConnected() drop path. notifyDaxCreateFailed clears createPending and, if the channel is still held, arms a kDaxCreateRetryMs retry — so an error reply or a connect-gap drop no longer leaves holders>0 / streamId==0 / createPending=true terminal. The suggested fix landed essentially verbatim.

  2. RADE hold not following a mid-session dax= change (@ten9876 item 2). Resolved. activateRADE now wires m_radeDaxReconcileConn on the RADE slice's daxChannelChanged, acquiring the new channel before releasing the old, independent of m_daxBridge. deactivateRADE disconnects it and releases m_radeDaxChannel. The old-channel leak / new-channel-never-created path is closed.

Both were valid on the commit they were filed against; they're now false positives against current head.

Other things I checked and they're clean:

  • No dangling references to the deleted symbols (ownsDaxChannel, m_tciDaxStreamIds, m_radeDaxStreamId, scheduleDaxRxStreamRemoval, etc.) — only a stale-comment mention survives in TciServer.h:58, harmless.
  • The removal-timer path releases m_streamMutex before re-entering unregisterDaxStream() (non-recursive mutex — no deadlock), and daxStreamUnregistered fires exactly once since the entry is erased first.
  • sliceRemoved orphan-release wired for both Bridge and Tci holders; generation-counter cancellation of in-flight deferred lambdas is consistent across acquire/release/reset.
  • Conventions honored: AppSettings (not QSettings) for TciPort, enum class : quint8, RAII connections, structured bindings. No new flat keys, TX path untouched.

Two minor, non-blocking notes:

  • Disconnected-retry spin. If a consumer holds a channel while isConnected() is false, the daxStreamCreateNeeded → notifyDaxCreateFailed → retry cycle self-loops every kDaxCreateRetryMs until connect or resetDaxChannelsForDisconnect() bumps the generation. It's low-frequency (2 s) and self-healing, so fine as-is — just flagging that onConnected reconciliation (which @ten9876 also suggested) would let it re-fire promptly rather than waiting up to one retry interval. Optional given the disconnect reset already clears the table.

  • Merge conflicts. GitHub reports mergeable_state: dirty and a merge-tree against main shows conflicts — this needs a rebase before it can land. Not a code issue, just housekeeping.

Nice work — the design and the test surface are both solid. 👍


🤖 aethersdr-agent · cost: $2.9256 · model: claude-opus-4-8

@jensenpat

Copy link
Copy Markdown
Collaborator Author

One follow-up refinement in e56de05a, found by adversarially re-tracing the retry logic against the protocol guide's captured behavior: the #1439 dax_clients nudge fired per create emission, so the new 2 s failure-retry cadence would also have re-asserted the slice mapping every cycle — and a same-value re-assert on a live binding provokes the radio's transient unbind/rebind dax=0/dax=<ch> pair. The manager's latches absorb that flap without emitting anything (no loop), but repeatedly sending a known transient-provoking command is exactly the shape this PR exists to eliminate. The nudge is now gated on the create's success reply — one nudge per confirmed stream, never per attempt.

💻 Generated with Claude Code (claude-fable-5 7/1/26) with architecture by @jensenpat

@jensenpat

jensenpat commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Final pass: wire-level deep dive on error codes, rebroadcast behavior, and firmware variance

Ran the deferred protocol captures against the FLEX-8400M (fw 4.2.18.41174) and re-ran the full live matrix on the current branch. Findings, and what they mean for this PR:

The transient-pair law (rebroadcast behavior, now precisely characterized)

A clean matrix over every slice set dax= transition × stream-presence combination:

Transition No stream Live stream
Fresh bind 0→N single dax=N single + stream rebind
Same-value N→N pair dax=0,dax=N pair + REAL stream unbind/rebind
Switch N→M single dax=M single + old-stream unbind
Unbind N→0 single single + unbind

The pair fires if the commanded value equals the current value — and on a live stream it's a genuine momentary unbind (audio interruption), not just status noise. The #4009 log (4.2.20) matches exactly: no firmware variance on the re-assert trigger. The 8600M's create-triggered rebroadcast (#3626) did not reproduce on the 8400M — that one is model/firmware-specific.

Stream-create error codes (the retry path's reality check)

  • stream create type=dax_rx dax_channel=<anything>no validation: 0, 5, 9, even −1 all mint real streams. Malformed forms → 0x50000016 "Malformed command". DAX IQ does validate (0x50000066). slice set dax=90x50000016.
  • No slot exhaustion: 24 concurrent dax_rx streams on one channel, all accepted. Two dax_tx streams coexist.
  • Net: a well-formed dax_rx create effectively cannot fail on current firmware — the notifyDaxCreateFailed retry path is defense-in-depth for connect-gap drops and future firmware, not a hot path. (It could not be exercised live for exactly this reason.)

One more refinement this surfaced (587728ad)

The law makes the #1439 nudge — even success-gated — a same-value re-assert that would self-inflict one pair plus a real stream blip per create. Since statuses precede the reply (guide §1.3), the registration status has already told us whether firmware auto-bound (slice=<letter>, captured on ≥4.2.18). The nudge is now skipped when binding was observed — a true legacy-only fallback. Result, live-proven on this branch: a full TCI audio bring-up emits exactly ONE command (stream create) and ZERO slice set dax re-asserts. The entire matrix session (TCI bring-up, bridge co-hold on/off, out-of-band slice add/remove exercising the new sliceRemoved hooks end-to-end) produced one genuine auto-assign slice set and three stream create/removes — the theoretical minimum.

Firmware-variance verdict

Storm immunity here is structural, not tuned: commands are emitted only at monotone holder transitions (first-holder create, grace-window last-holder remove, latched retries); echo paths move holder bits only, so whatever a firmware rebroadcasts — on re-assert (all captured firmware), on create (8600M), or something new — echoes cannot re-emit commands and no storm/resubscribe loop is constructible. The grace window (1.5 s) exceeds the captured pair spacing (~80 ms) by ~20×. All of this, including the transition matrix and the validation findings, is going into docs/architecture/flex-protocol/state-machines.md (§7.4b/§7.5b/§7.7 + error-code table) in the docs PR.

💻 Generated with Claude Code (claude-fable-5 7/1/26) with architecture by @jensenpat

jensenpat and others added 7 commits July 4, 2026 16:15
…hersdr#3305)

Replace three hand-rolled, mutually-peeking DAX ownership trackers (DAX
bridge m_daxSliceLastCh teardown logic, TCI m_tciDaxStreamIds/
m_tciDaxBorrowedChannels, RADE m_radeDaxStreamId) with refcounted
acquire/releaseDaxChannel(channel, consumer) in PanadapterStream — the
single decider of when a radio-side dax_rx stream exists. RadioModel is
the command plane (stream create/remove + a one-shot aethersdr#1439 dax_clients
nudge tied to the create) and the single, client_handle-filtered stream
status registrar (RADE's old private hook had no handle filter and could
adopt a foreign client's broadcast stream status).

Structural consequences:
- No commands are ever emitted from status-echo edges. The radio answers
  a re-assert of a live dax binding with a transient unbind/rebind
  dax=0/dax=<ch> pair (flex-protocol/state-machines.md §7.4); the old
  unconditional re-asserts in onDaxChannelChanged/ensureDaxForTci turned
  that into the ~12-15 Hz slice-set storm of aethersdr#4009 and the create/remove
  churn of aethersdr#3626. Both are now impossible by construction.
- Last-holder removal runs behind a 1.5 s grace window (absorbs the
  transient pairs; re-acquire cancels), subsuming aethersdr#3626's deferred
  removal and aethersdr#2895's cross-consumer guards.
- Radio-side stream death while held (profile load slice teardown)
  auto-recreates after a backoff — the aethersdr#3476 re-arm, now universal.
- Disconnect resets the table with no commands (radio reaps its own
  streams); slice removal releases the bridge's hold (orphan fix).

Live-proven against FLEX-8400M fw 4.2.18 as a second MultiFlex station:
TCI audio_start -> exactly one stream create + one-shot re-assert, 0
storm commands; DAX bridge toggled on+off across a live TCI stream ->
co-hold/release with no stream remove and uninterrupted audio (the
aethersdr#3363/aethersdr#2886 class); last holder release -> grace -> single stream
remove; clean session release on exit.

Closes aethersdr#3305. Fixes aethersdr#4009. Supersedes the tracker-patch approach of
PR aethersdr#4010 (which traded the storm for off->on retoggle and restored-slice
silence regressions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ecycle testing

- `get dax`: snapshot of the aethersdr#3305 centralized DAX RX channel-ownership
  table (per-channel holders, stream id, create-pending) plus each
  slice's dax assignment — the direct assertion surface for storm
  regressions (aethersdr#4009), co-hold survival (aethersdr#3363), and grace-window
  removal, replacing log-grepping.
- `tci start|status|stop [abrupt]`: in-process TCI client simulator
  speaking the WSJT-X dialect (init burst -> ready -> audio_samplerate +
  audio_start, binary-frame counting). `stop abrupt` reproduces the
  WSJT-X watchdog-reconnect shape for teardown/reap tests. Removes the
  external-WebSocket-client dependency from TCI test plans.

Both verbs live-proven against FLEX-8400M: tci start -> ~47 frames/s,
get dax shows holders=[tci] with the live stream id; tci stop abrupt ->
TCI debounce -> release -> 1.5s grace -> single stream remove ->
get dax channels drain to empty.

Documented in docs/automation-bridge.md (verb catalog kept complete per
the aethersdr#3928 audit) and tools/automation_probe.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… RADE/TCI hold reconciliation, sim teardown

Review items from @ten9876 (changes requested) and @NF0T's independent
confirmation pass:

1. (blocking) stream-create failure no longer wedges the channel:
   RadioModel sends the create via sendCmd with a response callback and
   reports nonzero replies (and connect-gap drops) to the new
   PanadapterStream::notifyDaxCreateFailed(), which clears the
   createPending latch and — while the channel stays held — retries on a
   2 s cadence. A slots-exhausted radio costs one command per 2 s and
   heals the moment a slot frees; the aethersdr#3669 wedge class stays dead on
   the first-create path too.
2. RADE's hold now follows mid-session dax= changes on its slice via a
   dedicated daxChannelChanged reconciler (independent of the
   Bridge-only reconciler, present on Windows builds too); disconnected
   at deactivateRADE.
3. Release-during-inflight-create no longer bounces create->remove
   churn: the last-release path keeps the entry when a create is in
   flight, so registerDaxStream lands on it and schedules ONE
   deterministic removal.
4. TCI now releases its channel hold when the backing slice is removed
   (pre-existing orphan, closed here as the review suggested): a
   sliceRemoved hook releases any Tci-held channel no remaining slice
   carries and invalidates the trx routing cache.
5. `tci` sim tears itself down on a server-side close (running=false,
   restart accepted) instead of zombie-blocking `tci start`; guarded
   against double-teardown with the stop path.
6. scheduleDaxRecreateLocked keeps the mutation=>generation-bump
   invariant.
7. Dropped the dead daxStreamRegistered signal and fixed the header
   comment (TCI wires daxStreamUnregistered only).

Also aligned the docs/automation-bridge.md tci example with the real
default port (TciPort setting, 50001).

Verified: clean build; ctest 53/54 (theme_manager_test pre-existing/
environmental); sim-zombie fix proven live radio-free (server-side close
-> running=false + closeReason, immediate restart accepted). Radio was
In_Use during this pass, so the storm/co-hold scenarios were not re-run;
they are unchanged paths plus the guarded edge fixes above.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while adversarially tracing the retry logic against the protocol
guide: the nudge fired per create EMISSION, so the 2 s failure-retry
cadence would also re-assert the slice mapping every cycle — and a
same-value re-assert on a live binding provokes the radio's transient
unbind/rebind dax=0/dax=<ch> pair (state-machines.md §7.4). The
manager's latches absorb that flap commandlessly (no loop), but
repeatedly emitting a known transient-provoking command is exactly the
shape aethersdr#3305 exists to eliminate. Now: one nudge per CONFIRMED stream,
inside the create's success reply callback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stream

Live wire matrix on fw 4.2.18 (guide captures cap6/cap6b) established
the transient-pair law: a same-value `slice set dax=` re-assert — and
ONLY a same-value re-assert — makes the radio emit the transient
dax=0/dax=<ch> pair AND momentarily unbind/rebind a live stream (a real
audio interruption). Since statuses precede the create reply, the
registration status has already told us (slice=<letter>) whether the
firmware auto-bound stream<->slice — captured behavior on >=4.2.18. The
nudge now fires only when the registration showed NO binding: a true
legacy-firmware fallback that never self-inflicts the pair.

Proven live: full TCI bring-up now emits exactly ONE command
(stream create) and ZERO slice-set re-asserts; the whole matrix session
(TCI + bridge co-hold toggles + out-of-band slice add/remove) produced
one genuine auto-assign slice-set and three stream create/removes total.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sim teardown fix (aethersdr#4017 item 5) added a QWebSocket::disconnected lambda
that does m_tciSim->deleteLater()+null when sender()==m_tciSim. But the stop
path nulls m_tciSim AFTER abort()/close(). abort() emits disconnected
synchronously (same-thread direct delivery), so it re-enters that lambda while
m_tciSim is still set → the lambda nulls it, and the stop path's subsequent
m_tciSim->deleteLater() fires on a dangling/null pointer. Reachable via the
`tci stop abrupt` verb this PR ships for lifecycle testing. (The graceful
close() path defers disconnected, so it was safe by accident.)

Capture the socket into a local, null m_tciSim (and the ready/audio flags)
FIRST — so the synchronous lambda's sock==m_tciSim guard fails and no-ops —
then abort()/close() and deleteLater() on the local. Preserve the graceful
audio_stop by snapshotting m_tciSimAudioStarted before the reset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nown, not in the create reply

The nudge-skip gate read a channel-scoped m_daxRxBoundObserved bool set from
whichever dax_rx registration status preceded the create reply. That couples
two async events through a per-channel latch that (a) isn't correlated to the
specific create's generation like the rest of the subsystem and (b) is reset
only on disconnect. When the auto-bind slice=<letter> status arrives AFTER the
create reply — reachable on the WAN/SmartLink transport (outside the ≥4.2.18
LAN capture) or any firmware that binds after replying — the gate defaults to
false and fires a same-value `slice set dax=` re-assert: the transient
dax=0/dax=<ch> pair + stream unbind/rebind audio blip the gate exists to avoid.

Move the decision into handleDaxRxStreamRegistry, which runs exactly when the
binding state is definitive (slice= populated or not) and is already filtered
to streams we own. Nudge only when the radio did NOT auto-bind AND a slice
carries the channel. Race-free across transports; zero behavior change on the
captured LAN path (auto-bound → skip). Removes the now-unnecessary
m_daxRxBoundObserved latch and its disconnect-time clear.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ten9876 ten9876 force-pushed the fix/3305-dax-stream-refcount branch from 587728a to dbe5238 Compare July 4, 2026 23:21

@ten9876 ten9876 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (2nd pass): approve

Re-reviewed against the rebased head. Every blocking item from the first round is resolved, and a fresh adversarial sweep of the fix commits surfaced two new issues that are now fixed. Rebased onto current main and pushed the fixes directly (the branch was CONFLICTING).

First-round issues — all resolved (verified against code, not just commit messages)

# Issue Resolution
1 stream create fire-and-forget wedge sendCmd reply callback -> notifyDaxCreateFailed + kDaxCreateRetryMs retry; also on the !isConnected() drop
2 RADE hold not following mid-session dax= activateRADE wires daxChannelChanged, acquire-new-before-release-old, independent of m_daxBridge
3 create->remove churn on last-release-during-create releaseDaxChannel keeps the entry so registerDaxStream schedules one deterministic removal
4 TCI sliceRemoved orphan Tci hold released on slice removal (slice erased from m_slices before sliceRemoved, so the stillWanted scan is correct)
5 TCI sim zombie after server-side close disconnected lambda tears down with a sender()==m_tciSim guard
6 missing generation bump in recreate scheduleDaxRecreateLocked now pairs createPending=true with ++m_daxGenCounter
7 dead daxStreamRegistered signal removed entirely (zero refs tree-wide)

Refcount core re-swept clean: no emit-under-lock, single daxStreamUnregistered per teardown, no recursive-mutex re-entry, channel bounds enforced, generation cancellation consistent.

Two new findings from this pass — fixed and pushed

A. tci stop abrupt null-derefs m_tciSim (a regression introduced by the item-5 fix). abort() emits QWebSocket::disconnected synchronously, re-entering the teardown lambda while m_tciSim was still set; the stop path's following deleteLater() then fired on a dangling pointer — reachable via the tci stop abrupt verb this PR ships. Fix 5285f549: null m_tciSim (and snapshot the audio flag) before abort()/close() so the lambda's guard no-ops; graceful audio_stop preserved.

B. #1439 nudge over-fires on WAN/SmartLink. The nudge-skip gate read a channel-scoped latch set by whichever registration status preceded the create reply — not correlated to the create's generation, reset only on disconnect. When the auto-bind slice= status arrives after the reply (reachable on the WAN transport, outside the >=4.2.18 LAN capture), the gate defaulted false and fired a same-value slice set dax= re-assert — the transient dax=0/dax=<ch> + audio blip it exists to prevent. Fix dbe5238f: decide the nudge in handleDaxRxStreamRegistry, where the binding is definitively known and the path is already filtered to our streams. Race-free across transports, zero behavior change on the captured LAN path; the racy latch is deleted.

Both build clean (AetherSDR, no new warnings).

Left as non-blocking follow-ups

  • Disconnected-retry self-loop (0.5 Hz) if a channel is acquired inside the disconnected window — bounded and self-heals on reconnect.
  • notifyDaxCreateFailed reply callback isn't generation-guarded — not exploitable today (pending callbacks aren't flushed on disconnect); cheap future hardening.

Excellent refactor — collapsing three mutually-peeking ownership trackers into one refcounted authority with the "no commands from status-echo edges" invariant is the right structural fix for the #4009 storm class. Approving.

@ten9876 ten9876 enabled auto-merge (squash) July 4, 2026 23:33
@ten9876 ten9876 merged commit e9d3c82 into aethersdr:main Jul 5, 2026
6 checks passed
NF0T added a commit to NF0T/AetherSDR that referenced this pull request Jul 5, 2026
…thersdr#176)

Adversarial review of the C-QUAM branch found that CquamDsp divided
already-normalized DAX IQ samples by 32768 a second time, pushing real
signal levels below the fixed epsilon gates used for L-R extraction and
pilot lock — pilot lock was effectively unreachable for any real signal,
which is the likely actual cause of the "no lock on any station" result
reported when testing over the air, not station scarcity. Fixes:

- Remove the erroneous /32768 rescale; make the L-R gate and its epsilon
  relative to the tracked carrier envelope (dcEnv) instead of an absolute
  constant, so detection no longer depends on the input's absolute scale.
  kAudioGain/AGC (the existing AM-level-matching work) are untouched —
  they already normalize by dcEnv the same way, so they were never
  affected by the scale bug.
- Add DaxIqModel::acquireChannel/releaseChannel (named to match
  PanadapterStream's acquireDaxChannel/DaxConsumer convention from
  aethersdr#3305/aethersdr#4017) so CquamDemodulator and WfmDemodulator can't silently
  collide on the same hardcoded DAX IQ channel; MainWindow also
  cross-checks m_wfmSliceId/m_cquamSliceId for WFM's Windows capture
  path, which bypasses DaxIqModel entirely.
- activateCQUAM() now commits SliceModel's enabled state and emits
  cquamEnabledChanged only after the demodulator is confirmed active,
  removing a duplicate emit and a stuck-UI-on-failure path.
- Remove an unrelated stray VfoWidget::setFrequencyHovered declaration
  with no implementation or call sites.
- Fix stale doc comments (wrong sample rate, wrong "bypasses" claim,
  wrong dB value) and a constant-name collision between
  CquamDemodulator and CquamDsp.
- Add tests/cquam_dsp_test.cpp: synthesizes a real C-QUAM composite
  signal and verifies pilot lock, stereo separation, mono fallback, and
  output level with no radio or over-the-air signal needed. Verified the
  test fails on the original bug and passes on the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: medium Medium priority

Projects

None yet

3 participants