Skip to content

Agent automation bridge: radio-side display-stream inventory + leak detection (streams verb) — prevent leaked/duplicate pan & waterfall streams #3856

Description

@jensenpat

Summary

Add a radio-side display-stream inventory + leak-detection capability to the agent automation bridge (and, ideally, a standing in-app audit) so we can directly observe and prevent a whole class of bugs: leaked panadapter/waterfall streams left allocated on the radio, duplicate pans after reconnect/Multi-Flex races, and lingering streams from a crashed or botched teardown.

Today the client's view (get pans, m_panadapters) is cleaned up on the radio's removed echo, so it always looks clean even when the radio still holds a stream. We have no way to ask "what does the radio actually have allocated, and does it match what we own?" That blind spot is exactly what hid #3843, and it's the same blind spot behind the "leaving pans on the radio" and "duplicate pans during races" reports.

Background

Split out of #3855 (fixes #3843). While fixing the panafall close path I prototyped a streams bridge verb + a UDP-orphan detector and proved the close-path fix with it. I removed that prototype from the bug-fix PR to keep it minimal and low-risk — this issue tracks designing and landing it properly as its own feature.

Key empirical finding from that work (informs the whole design): on a live FLEX-8400M, firmware 4.2.18, the radio stops the waterfall UDP stream when the panadapter is removed — so a waterfall leaked by a missing display panafall remove is a resource left allocated on the radio, not continued UDP traffic. A packet-based detector therefore cannot see it on this firmware; we need a radio-authoritative resource query. (Continued-UDP leaks do happen on older 6000-series firmware — see #268 "waterfall continues to scroll".)

Proposed feature — two complementary layers

Layer A — UDP-orphan detector + streams verb (catches continued-UDP leaks, #268-class)

PanadapterStream records any display packet (PCC_FFT 0x8003 / PCC_WATERFALL 0x8004) whose stream id was ever registered this session AND is no longer registered — i.e. a stream the radio is still transmitting that we've let go of.

Bridge surface:

  • streams{ registeredPanStreams[], registeredWfStreams[], orphanStreams[], orphanCount }, where each orphan is { streamId, kind: "waterfall"|"panadapter", packets, age_ms }.
  • streams reset → clear the orphan tally to re-baseline a before/after measurement.

Read semantics: an orphan whose packets climbs across reads with small age_ms is a live leak; one that stops growing was a brief in-flight tail.

Design gotchas already discovered (adversarial review of the prototype) — bake these in:

  • Gate the orphan check on ever-registered ∧ not-currently-registered, not on "live known-set non-empty." The naive !isEmpty() gate has two bugs: (1) it goes blind after pan close all (live set empties → a real leak reads 0), and (2) it false-positives a freshly-created 2nd pan's waterfall during its sub-second registration-lag window.
  • Hot-path safety: only touch the orphan map on the (rare) orphan branch, under the existing m_streamMutex; bound the map (e.g. 32 entries); clear it (and the ever-registered sets) on disconnect.

Layer B — radio-authoritative display-resource query (catches resource-level lingering, #3843-class on modern firmware) — the missing piece

A verb that asks the radio for its authoritative set of display objects and diffs it against what the client owns:

  • Re-subscribe / refresh (sub display all) and collect the radio's display pan 0x… … / display waterfall 0x… … status echoes within a short window; and/or read capacity counters if the radio exposes them (available_panadapters / available_waterfalls).
  • Attribute each radio-side object by client_handle: ours, foreign (another Multi-Flex client), or orphan (no owner / not in our model) → an orphan = a leaked or crashed-session stream.
  • Report { radioPanCount, radioWfCount, ours[], foreign[], orphan[] }.

This is what actually detects the #3843 waterfall resource leak on fw 4.2.18, and it generalizes to detecting duplicate pans (radio has 2 where the client expects 1) and lingering pans from a prior crashed session.

Proposed surface: streams radio (or streams inventory).

Preventing the bug classes (the point of the feature)

  1. Leaked waterfall/pan after botched teardown — Layer A (continued-UDP) + Layer B (resource-level). Guards Closing a panafall-created panadapter never removes the waterfall stream (display panafall remove missing); removePanadapter uses non-existent 'display pan close' #3843 and Multi-panadapter: 2nd panadapter loses FFT spectrum (waterfall continues), cannot be closed — persistent across reboot and sole-client reconnect #268.
  2. Duplicate pans on reconnect / Multi-Flex races — Layer B run at the end of the reconnect sequence (ties into the [codex] Fix panadapter restore across reconnects #3429 stage/reclaim/prune flow): if the radio reports a pan we don't own / a duplicate, surface it and optionally auto-reconcile by pruning the orphan through the (now-correct) RadioModel::removePanadapter.
  3. Lingering pans from a crashed prior session — Layer B on connect: any unowned radio-side display object is a candidate for cleanup.
  4. Optional standing audit — a periodic in-app (debug/automation-gated) check that warns when radio-side display resources exceed client-owned: a continuous leak alarm rather than a manual probe.

Acceptance criteria

Implementation notes / references

  • The aether.protocol log category (log set aether.protocol on, then log tailevents[]) and the on-disk session log at ~/Library/Preferences/AetherSDR/logs/aethersdr-<ts>.log are the proof channels for command-level verification (RadioModel::sendCommand: lines).
  • sub display all re-subscribe must not disrupt live rendering; collect within a bounded window and attribute by client_handle.
  • The Layer-A detector prototype (with the ever-registered fix) is in the force-pushed history of PR fix(spectrum): free the waterfall stream when closing a panafall (#3843) #3855 if useful as a starting point.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    New FeatureNew feature requestmaintainer-reviewRequires maintainer review before any action is takenmulti-panMulti-panadapter layout and slicingprotocolSmartSDR protocolspectrumPanadapter and waterfall

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions