feat(automation): in-app agent-drivable test bridge for the GUI (#3646)#3710
Conversation
… Phase 0) Adds the first slice of the agent-drivable test surface proposed in aethersdr#3646: an in-process automation channel for cross-OS UX testing of the panadapter/waterfall and applet controls, gated behind AETHER_AUTOMATION and off in production. AutomationServer is a QLocalServer speaking newline-delimited JSON with two read-only verbs (plus ping): - dumpTree → ARIA-style JSON snapshot of every top-level QWidget hierarchy (class, objectName, accessibleName, enabled, visible, global geometry, and a best-effort scalar/text value for sliders, buttons, combos, spin boxes, progress bars, and labels). This is our "DOM snapshot" for controls. - grab <target> [path] → PNG capture of a single widget, resolved by objectName, class name, or accessibleName. The QRhi panadapter is routed through QRhiWidget::grabFramebuffer() (QWidget::grab() returns an empty surface for a GPU widget), so the live spectrum is captured correctly. Kept as a dedicated channel rather than overloading TCI, per the issue's recommendation: TCI has external protocol-compat constraints and test verbs must not leak into a radio-control protocol. Wiring: - main.cpp constructs the server after window.show() only when AETHER_AUTOMATION is set; AETHER_AUTOMATION_SOCKET overrides the default QLocalServer name. - New lcAutomation logging category, registered in the Support dialog. - On listen, the resolved socket path is written to a discovery file (<temp>/aethersdr-automation.json) so a driver can find the endpoint without guessing the platform-specific socket/pipe. start() always clears a stale socket (removeServer) and rewrites the discovery file, so an abnormal exit self-heals on the next launch. tools/automation_probe.py is a dependency-free (no Qt) driver that talks to the AF_UNIX socket / Windows named pipe directly. Its demo mode produces the Phase-0 deliverables: an applet semantic snapshot (tree.json) and a panadapter PNG. Verified live on macOS (Qt 6.11, GPU spectrum path): ping/dumpTree/grab all succeed; dumpTree captured 43 top-level windows (196 accessibleNames, 1212 live values) and grab produced a 2896x1502 framebuffer readback of the live panadapter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds docs/automation-bridge.md — an AI-agent-audience reference for the Phase 0 automation bridge: when to use it, the QLocalServer + JSON contract, per-verb request/response schemas (ping, dumpTree, grab), widget-targeting rules, snapshot→act→assert recipes, gotchas (GPU grabFramebuffer, off-by- default, stale-socket self-heal, offscreen for CI), and the aethersdr#3646 roadmap. Hooks it from AGENTS.md (the canonical guide every agent reads — Claude via CLAUDE.md, Codex directly) with a short "verify the GUI without pixels" section next to the Accessibility rules, matching the repo's duplication-by-pointer convention and the COMMIT-SIGNING.md "AI assistants read this first" callout style. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…3646 Phase 1) Adds the Phase 1 "drive + assert" half of the agent automation bridge: - invoke <target> <action> [value] — drive a control deterministically: click / toggle / setChecked / setValue / setText / setCurrentText / setCurrentIndex. Resolves the target exactly like grab (objectName → class → accessibleName) and echoes the post-action state as `newValue` for a free round-trip assertion. - get <model> [selector] [property] — live JSON snapshot of model truth: radio | slice <id|active|tx> | slices | pan <panId|active> | pans. With a trailing property name, returns just that field. Lets a scenario assert on state (frequency, mode, filter, NB/NR, center MHz, min/max dBm, …) without screenshots. Snapshots are hand-built from existing getters, so no model has to be re-annotated as Q_PROPERTY and the diff touches no model internals. TX SAFETY (issue requirement): invoke refuses any control whose objectName, accessibleName, or button text looks transmit-related — mox, ptt, tune (incl. the ATU tune button, which emits a carrier), transmit, vox — and returns a blocked error without ever calling the widget. Override only via AETHER_AUTOMATION_ALLOW_TX=1 for deliberate hardware-in-the-loop tests. A test bridge must never key a live radio by accident; over-blocking is the safe failure mode. Wiring: - MainWindow gains a public radioModel() accessor (active-session model, tracks Multi-Flex switches); main.cpp passes it to the server via setRadioModel() so get() has a live handle. - handleLine() parses the richer request shape (action/value/model/ selector/property) for both JSON and bare-line forms. - tools/automation_probe.py gains `invoke` and `get` subcommands. - docs/automation-bridge.md + AGENTS.md document both verbs, the action/ model tables, and the TX-safety contract. Verified live against a connected FLEX-8400M: get radio/slice/pan return correct state; invoke setValue on the master-volume slider round-trips; and the TX guard refused all four transmit controls (MOX, Tune, ATU tune, VOX) with the radio staying transmitting=false throughout. The a11y-linter backlog (the issue's other Phase 1 item) is split into a dedicated follow-up so it can be reviewed as an accessibility change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hersdr#3646) Two QA-driven improvements to the automation bridge, found by dogfooding it against a live FLEX-8400M: 1. get transmit — expose TransmitModel as a model snapshot (RF/tune power, mic/processor/monitor, VOX/AM/DEXP, TX filter, CW speed/pitch/break-in/ delay/sidetone/iambic/monitor, ATU, APD). Previously get only reached RadioModel/SliceModel/PanadapterModel, so TX/Phone/CW applet controls could only be validated at the widget level; now a scenario can assert the change reached the radio model. Keying state (mox/tune/transmitting) is reported read-only and is never driven from here. 2. TX-safety guard scoped to buttons — only a discrete button action (click/toggle) can key the radio (MOX/PTT/TUNE/ATU-tune/VOX enable are all QAbstractButtons), so the guard now ignores sliders/combos/spinboxes. Setpoint controls like "Tune power", "RF power", and "VOX level" were being over-blocked even though moving a value setter can't transmit; they are now drivable while every keying button stays blocked. Safety is unchanged — invoke still cannot key through a value setter. Verified live (no transmission): get transmit returns 35 fields; setting RF power / mic gain / CW speed / DEXP now validates at the model level; "Tune power" and "VOX level" sliders are drivable; MOX, Tune, ATU tune, and VOX enable buttons remain blocked; radio stayed transmitting=false throughout. The same dogfooding surfaced an unrelated app bug (TransmitModel:: setAmCarrierLevel omits the optimistic local update its sibling setters have, so amCarrierLevel() reads stale until the radio echoes) — filed separately to keep this change focused on the bridge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the "verify the GUI without pixels" pointer to AGENTS.md — the canonical guide every agent on this repo reads (Claude via CLAUDE.md, Codex directly) — so an agent discovers the automation bridge while working the tree. Summarizes the four verbs (dumpTree/grab/invoke/get), the button-scoped TX-safety guard, and the get models (incl. transmit), and links the full reference at docs/automation-bridge.md. Included at maintainer request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ten9876
left a comment
There was a problem hiding this comment.
Foundational, cleanly-scoped work — purely additive, genuinely inert in production (confirmed: the server is never constructed when AETHER_AUTOMATION is unset), no model internals touched, cross-platform via QLocalServer, and the QRhiWidget::grabFramebuffer() capture path is exactly right. I want this in. Requesting changes on one thing: the TX-safety guard, which is the bridge's headline guarantee and currently has real holes.
Blocking: isTransmitControl() misses TX-keying controls
The guard blocks QAbstractButtons whose name/text contains mox/ptt/tune/transmit/vox. The PR's verification covered MOX/Tune/ATU/VOX — but auditing the actual keying buttons in the GUI against those keywords, two genuinely key the transmitter and are not blocked, because they're labeled "Send":
src/gui/CwxPanel.cpp:172—QPushButton("Send")sends CW → keys TX. "send" matches no deny term, soinvoke … clickwould transmit against a live radio.src/gui/Ax25HfPacketDecodeDialog.cpp:2608— the terminal"Send"button transmits an AX.25 packet → keys TX. Also unguarded. (The dialog's primary"Transmit"button at :940 is caught — good.)
This is exactly the failure mode the guard exists to prevent (an agent keying a live transmitter during hardware-in-the-loop tests). The blocklist also over-blocks harmlessly — src/gui/MemoryDialog.cpp:432 "Tune" is a frequency-recall button, not a carrier (fail-safe direction, fine) — and the three "TX" buttons (EqApplet.cpp:188, AetherialAudioStrip.cpp:221, ClientChainApplet.cpp:101) look like RX/TX editor toggles rather than keying; a one-line confirmation would be good.
Requested change
The substring blocklist is structurally fragile: any TX-capable control whose name lacks a magic word is silently unguarded, and new keying controls added later inherit no protection. Please move to a positive marker the guard checks authoritatively — e.g. a dynamic property setProperty("aetherTxKeying", true) set on the genuinely-keying controls (MOX, PTT, Tune, ATU, CWX Send, packet Send), with isTransmitControl() honoring that property (keep the name heuristic as a belt-and-suspenders fallback if you like). At minimum, the CW-send and packet-send paths must be covered before this lands, since the guard is the safety story for running automation against real hardware.
Non-blocking (worth doing in the same pass)
- Socket permissions.
m_server->listen()is called withoutsetSocketOptions(QLocalServer::UserAccessOption). For a TX-reachable socket whose path is advertised in a shared-temp discovery file, restricting to the owning user (0600) is the safer default. Cheap, and it closes "another local user drives my GUI." - RFC #3646 carries the
rfclabel and is still open — worth formally resolving it alongside this merge since it's the sanctioned RFC for the feature area.
Everything else looks great. Once the keying-control coverage is authoritative (positive marker incl. CW/packet send), I'm a yes.
Addresses the maintainer's blocking review: the substring blocklist was
structurally fragile — any TX-capable control whose label lacked a magic word
was silently unguarded, and the audit found two real keying buttons labeled
"Send" (CWX CW send, AX.25 packet send) that would have transmitted against a
live radio under `invoke … click`.
Replace the name-matching primary mechanism with a positive, authoritative
marker:
- New src/core/TxKeyingMarker.h: markTxKeying(w) sets the "aetherTxKeying"
dynamic property at a keying control's creation site. isTransmitControl()
honors that property first — a control is blocked because it was *declared*
keying, not because its name happened to match.
- Marked every genuinely-keying control found by auditing the keying call
sites (setMox/startTune/atuStart/requestPttOn/send): TxApplet TUNE/MOX/ATU;
CwxPanel "Send"; Ax25 "Transmit", terminal "Send", "Beacon Now", "Send APRS
Msg"; AtuPreTuneDialog START / "Tune this frequency" / "Continue". The two
"Send" buttons and the APRS beacon/message buttons match no keyword and were
previously unguarded.
- The button-scoped name heuristic stays only as a logged belt-and-suspenders
fallback (now also covers cwx/atu); when it fires it warns that the control
needs an explicit markTxKeying().
- dumpTree now reports "keying": true on marked controls so an agent can see
what invoke() will refuse before trying.
- resolveWidget gains a last-resort button-text tier so a control known only
by its label ("Send", "Transmit") is targetable.
Also from the review (non-blocking): harden the socket with
setSocketOptions(QLocalServer::UserAccessOption) so the TX-reachable endpoint
is owner-only (0600), and confirm the three "TX" buttons (EqApplet,
AetherialAudioStrip, ClientChainApplet) are RX/TX editor toggles, not keying —
left unmarked.
Verified live (no transmission): the CWX "Send" button (matches no keyword) is
now blocked via the marker; MOX/Tune/ATU blocked; "Tune power"/"RF power"
sliders drivable; radio stayed transmitting=false throughout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the thorough review, @ten9876 — the substring-fragility call was right, and the audit turned up more than the two Blocking — TX guard is now marker-driven, not name-drivenReplaced the substring blocklist as the primary mechanism with a positive, authoritative marker:
Verified live (no transmission): the CWX
Confirmed — all three are checkable editor/view toggles, not keying, so left unmarked: Non-blocking
|
ten9876
left a comment
There was a problem hiding this comment.
Re-reviewed cef03f2b — every concern resolved, and the fix is better than what I asked for. Clearing my changes-requested.
TX guard is now marker-driven (verified against the diff):
isTransmitControl()checks theaetherTxKeyingproperty first (authoritative), before any name heuristic — a control is blocked because it was declared keying, not because its label matched a word. ✓- Every genuinely-keying control is marked at its creation site: CWX
Sendand packetSend(the two I flagged), plus Beacon Now and Send APRS Msg (which you found — same failure mode, I'd missed them), plus TUNE/MOX/ATU and the AtuPreTuneDialog per-point wizard buttons. Confirmed eachmarkTxKeying(m_…)call site, and that every caller#includesTxKeyingMarker.h. ✓ - Name heuristic retained as a logged belt-and-suspenders fallback that warns the control needs an explicit
markTxKeying(). Good defense-in-depth. ✓ dumpTreereports"keying": trueon marked controls, so an agent sees whatinvoke()will refuse before trying. Nice touch. ✓- The three
"TX"buttons confirmed as non-keying editor/view toggles, correctly left unmarked. ✓
Non-blocking, both done:
- Socket
setSocketOptions(QLocalServer::UserAccessOption)beforelisten()— owner-only endpoint. ✓ - RFC #3646 to be closed as the sanctioned RFC on merge. 👍
Approving. One gate left: CI is mid-run on this commit (build/macOS/Windows/analyze pending) — merge once those go green. Excellent work, and thanks for hunting down the extra two keying buttons.
There was a problem hiding this comment.
Thanks for this, @jensenpat — this is an exceptionally well-built and well-documented feature. The marker-based TX guard (markTxKeying() / aetherTxKeying) is the right design: declaring intent at the creation site is far more robust than a substring blocklist, and it correctly catches the "Send" / "Beacon Now" buttons no keyword would. CI is green across build/macOS/Windows/CodeQL/a11y, the change is cleanly scoped (purely additive, model internals untouched), error handling at the socket/JSON boundary is thorough, and lifetime/teardown ordering in main.cpp is correct (the automation unique_ptr destructs before window, so the QPointer<RadioModel> never dangles). Nice work.
One finding worth addressing before merge, plus a couple of minor notes.
dumpTree leaks password fields in cleartext (worth fixing)
widgetValue() returns le->text() for any QLineEdit (AutomationServer.cpp), with no check on echo mode. There are at least two QLineEdit::Password fields in the app:
src/gui/ConnectionPanel.cpp:407— radio connection passwordsrc/gui/MqttSettingsDialog.cpp:110— MQTT broker password
So dumpTree (and the tree.json that automation_probe.py demo writes to a temp file) will serialize those secrets in plaintext whenever the relevant dialog has been opened. The socket is same-user 0600 and the whole bridge is opt-in behind AETHER_AUTOMATION, so this isn't a privilege-escalation hole — but writing live credentials into a world-readable temp artifact (tree.json) is an easy footgun for an agent or CI run. Cheap fix in widgetValue():
if (auto* le = qobject_cast<const QLineEdit*>(w))
return le->echoMode() == QLineEdit::Normal ? le->text() : QStringLiteral("<hidden>");(That preserves assertability — an agent can still confirm the field is non-empty — without exfiltrating the value.)
Minor notes (non-blocking)
- Concurrent-instance socket clobber.
start()unconditionally callsQLocalServer::removeServer(serverName)beforelisten(). That's correct for the stale-socket self-heal you describe, but if a secondAETHER_AUTOMATION=1instance launches with the default socket name it silently steals the endpoint and rewrites the shared discovery file out from under the first. Edge case (two automation-enabled instances), and probably fine to leave — but a one-line log line noting the takeover, or appending the PID to the default socket name, would make it less surprising. grabwrites to a client-suppliedpath. Any path the running user can write is fair game. Same-user/opt-in makes this acceptable, just flagging it as intentional surface.
Neither minor note blocks merge; the password-field point is the one I'd like to see handled. Overall this is a clean, high-quality addition.
🤖 aethersdr-agent · cost: $4.5361 · model: claude-opus-4-8
…3717) Follow-up to #3710 (merged) — addresses @aethersdr-agent's post-merge review finding. ## Problem `dumpTree`'s `widgetValue()` returned `QLineEdit::text()` for **every** line edit, with no echo-mode check. So masked credential fields were serialized in **cleartext** — and the bridge writes `dumpTree` into a temp `tree.json` (`automation_probe.py demo`), so the secret lands in a temp-dir artifact an agent or CI run could read. Affected `QLineEdit::Password` fields: - `src/gui/ConnectionPanel.cpp:407` — SmartLink radio password - `src/gui/MqttSettingsDialog.cpp:110` — MQTT broker password The bridge is opt-in (`AETHER_AUTOMATION`) and the socket is same-user `0600`, so this isn't privilege escalation — but writing live credentials to a temp file is an easy footgun, and worth closing. ## Fix One spot in `widgetValue()` — redact any non-`Normal` echo mode (`Password` / `NoEcho` / `PasswordEchoOnEdit`): ```cpp if (auto* le = qobject_cast<const QLineEdit*>(w)) { if (le->echoMode() != QLineEdit::Normal) return le->text().isEmpty() ? QString() : QStringLiteral("<hidden>"); return le->text(); } ``` Slightly more than the suggested one-liner: a **populated** masked field serializes as `"<hidden>"`, an **empty** one omits `value` entirely — so the field stays assertable (present / empty / set) without ever exposing the secret. Normal fields are unchanged. ## Verification (live, FLEX-8400M) Typed a canary secret into the SmartLink password field via `invoke`, then: - password field → `"value": "<hidden>"` ✓ - canary string **absent** from the full `dumpTree` JSON ✓ - canary string **absent** from the `tree.json` artifact written by `automation_probe.py demo` ✓ - Normal `QLineEdit` (email) still serializes its value — no over-redaction ✓ - emptied masked field → `value` omitted ✓ 💻 Generated with Claude Code (Opus 4.8) with architecture by @jensenpat Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Summary
Adds the agent-drivable test surface proposed in #3646: an in-process, agent-first automation bridge for cross-OS UX testing of the panadapter/waterfall and applet controls. It gives an agent the "snapshot → act → assert" loop we already use for web work, but for our native Qt 6 Widgets UI — fast, deterministic, identical on macOS/Windows/Linux, and CI-friendly.
This closes a loop that agent-driven development otherwise leaves open, which is why Anthropic considers it foundational rather than nice-to-have. An agent like Claude Code can already write a feature, but on a native desktop app it has had no way to verify its own change: there is no DOM to query, no in-process handle on the accessibility tree, and a screenshot of a live SDR spectrum is non-deterministic noise that can't be asserted against.
Anthropic's view is that the single largest multiplier on agentic coding is a tight, machine-checkable feedback loop — an agent that can act, observe the real result, and assert on it produces dramatically fewer confident-but-wrong changes than one that edits blind and hands the work back for a human to test.
This bridge gives Claude Code (and any other agent working this repo) exactly that loop for AetherSDR's GUI: drive a control, read the model's ground truth, capture the panadapter, and prove the change did what was intended. It converts "this should work" into "I ran it and here is the evidence" — precisely the Evidence-Over-Assertion discipline the project constitution already asks of every contributor, human or AI. And the value compounds: the same verbs that let an agent self-check a one-off edit become the substrate for deterministic, cross-OS regression tests in CI, so a behavior verified once stays verified on every platform without a human in the loop.
The whole thing is gated behind
AETHER_AUTOMATIONand is completely inert in production: no server, no socket, no overhead when the env var is unset.This is a dedicated channel, not an extension of TCI — per the issue's explicit recommendation, since TCI has external protocol-compat constraints (eesdr-tci aborts on unknown commands) and test verbs must never leak into a radio-control protocol.
What's in it
A
QLocalServer(AF_UNIX socket / Windows named pipe) speaking newline-delimited JSON. Four verbs:dumpTreeQWidgettree — class, objectName, accessibleName, enabled, visible, global geometry, and a best-effort livevaluefor sliders/buttons/combos/labels. Our "DOM" for controls.grab <target> [path]QRhiWidget::grabFramebuffer()so the live spectrum is captured correctly (QWidget::grab()returns an empty surface for aQRhiWidget).invoke <target> <action> [value]click/toggle/setChecked/setValue/setText/setCurrentText/setCurrentIndex. Echoes post-action state asnewValue.get <model> [selector] [property]radio/transmit/slice/slices/pan/pans(frequency, mode, filter, NB/NR, center MHz, min/max dBm, RF/mic/CW TX-chain, …). Assert on state without screenshots.Targets resolve by
objectName→ class name →accessibleName→ button text. A driver discovers the socket via a discovery file (<temp>/aethersdr-automation.json), so it needs no Qt and no knowledge of the platform endpoint.🚨 TX safety
invokecannot key the radio by accident, and the mechanism is a positive marker, not name matching. Every genuinely-keying control is tagged at its creation site withmarkTxKeying()— theaetherTxKeyingdynamic property (seesrc/core/TxKeyingMarker.h) — covering MOX/PTT, TUNE, ATU, CWX CW send, and AX.25 packet/APRS send.invokerefuses anything carrying that marker, so a control is blocked because it was declared keying, not because its label happened to contain a magic word. That closes the structural hole a substring blocklist leaves: it catches keying buttons like "Send" and "Beacon Now" that no keyword would match, and a keying control added later is guarded the moment it's marked. Marked controls report"keying": trueindumpTree, so an agent sees what's off-limits before trying.A button-scoped name heuristic (
mox/ptt/tune/atu/transmit/vox/cwx) remains only as a logged belt-and-suspenders fallback for any keying control that predates the marker — when it fires it warns that the control needs an explicitmarkTxKeying(). Setpoint sliders/combos (Tune power,RF power,VOX level) are never blocked — moving a value setter can't transmit. Override is an explicit opt-in:AETHER_AUTOMATION_ALLOW_TX=1, for deliberate hardware-in-the-loop tests. The socket itself is restricted to the owning user (QLocalServer::UserAccessOption, 0600) so another local user can't drive the TX-reachable endpoint.Design notes
getuses hand-built model snapshots from existing getters rather than reflectingQ_PROPERTY— so the diff touches no model internals (PanadapterModel exposes its display state via plain getters, not properties) and one call returns the full assertable state an agent needs.MainWindow::radioModel()is the one new public accessor; it returns the active-session model sogettracks Multi-Flex session switches.main.cpphands it to the server viasetRadioModel().lcAutomationlogging category, registered in the Support dialog.removeServer) and rewrites the discovery file.Verification (live, on a connected FLEX-8400M)
dumpTree→ 43 top-level windows, 196 accessibleNames, 1212 live values.grab SpectrumWidget→ 2896×1502 GPU framebuffer readback of the live spectrum (not a blank).get radio/slice/pan→ correct live state (3.6 MHz LSB, center 3.892 MHz, −124/−34 dBm).invoke "Master volume" setValue 35→ round-tripped (newValue:35).Tune power,RF power) stayed drivable; radiotransmitting:falsethroughout.Scope
TxKeyingMarker.hguard helper, the agent guide, and the no-dependency Python driver. Small hooks elsewhere: CMake source list, a log category, theradioModel()accessor, themain.cppstartup gate, and one-linemarkTxKeying()calls at each keying control.docs/automation-bridge.md.Not in this PR (tracked in #3646)
setAccessibleName/QAccessibleInterfaceannotations the linter flags. Improves real accessibility anddumpTreesnapshot quality; handled as a dedicated accessibility change.QT_QPA_PLATFORM=offscreen+ agent scenarios + per-OS perceptual golden diffs.Test plan
💻 Generated with Claude Code (Opus 4.8) with architecture by @jensenpat