Skip to content

Commit ae61677

Browse files
gumadeirasgumadeiras
committed
test(qa-matrix): cover approval metadata scenarios
1 parent 795e58a commit ae61677

15 files changed

Lines changed: 1101 additions & 11 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Docs: https://docs.openclaw.ai
77
### Changes
88

99
- Dependencies: refresh provider and tooling dependencies, including AWS SDK, PI runtime packages, AJV, Feishu SDK, Anthropic SDK, tokenjuice, and native TypeScript/oxlint tooling. Thanks @dependabot.
10+
- Matrix/QA: add live Matrix approval scenarios for exec metadata, chunked fallback, plugin approvals, deny reactions, thread targeting, and `target: "both"` delivery, with redacted artifacts preserving safe approval summaries. Thanks @gumadeiras.
1011
- Codex: add Computer Use setup for Codex-mode agents, including `/codex computer-use status/install`, marketplace discovery, optional auto-install, and fail-closed MCP server checks before Codex-mode turns start. Fixes #72094. (#71842) Thanks @pash-openai.
1112
- Apps: consume Peekaboo 3.0.0-beta4 and ElevenLabsKit 0.1.1, align Swabble on Commander 0.2.2, and refresh macOS/iOS SwiftPM resolutions against the released dependency graph. Thanks @Blaizzy.
1213
- Plugin SDK: expose shared channel route normalization, parser-driven target resolution, raw-target compact keys, parsed-target types, and route comparison helpers through `openclaw/plugin-sdk/channel-route`, switch native approval origin matching onto that route contract with optional delivery and match-only target normalization, and retire the internal channel-route shim behind dated compatibility aliases for legacy key/comparable-target helpers. Thanks @vincentkoc.

‎docs/concepts/qa-matrix.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -64,15 +64,15 @@ Matrix QA does not accept `--credential-source` or `--credential-role`. The lane
6464

6565
The selected profile decides which scenarios run.
6666

67-
| Profile | Use it for |
68-
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
69-
| `all` (default) | Full catalog. Slow but exhaustive. |
70-
| `fast` | Release-gate subset that exercises the live transport contract: canary, mention gating, allowlist block, reply shape, restart resume, thread follow-up, thread isolation, reaction observation. |
71-
| `transport` | Transport-level threading, DM, room, autojoin, mention/allowlist scenarios. |
72-
| `media` | Image, audio, video, PDF, EPUB attachment coverage. |
73-
| `e2ee-smoke` | Minimum E2EE coverage — basic encrypted reply, thread follow-up, bootstrap success. |
74-
| `e2ee-deep` | Exhaustive E2EE state-loss, backup, key, and recovery scenarios. |
75-
| `e2ee-cli` | `openclaw matrix encryption setup` and `verify *` CLI scenarios driven through the QA harness. |
67+
| Profile | Use it for |
68+
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
69+
| `all` (default) | Full catalog. Slow but exhaustive. |
70+
| `fast` | Release-gate subset that exercises the live transport contract: canary, mention gating, allowlist block, reply shape, restart resume, thread follow-up, thread isolation, reaction observation, and exec approval metadata delivery. |
71+
| `transport` | Transport-level threading, DM, room, autojoin, mention/allowlist, approval, and reaction scenarios. |
72+
| `media` | Image, audio, video, PDF, EPUB attachment coverage. |
73+
| `e2ee-smoke` | Minimum E2EE coverage — basic encrypted reply, thread follow-up, bootstrap success. |
74+
| `e2ee-deep` | Exhaustive E2EE state-loss, backup, key, and recovery scenarios. |
75+
| `e2ee-cli` | `openclaw matrix encryption setup` and `verify *` CLI scenarios driven through the QA harness. |
7676

7777
The exact mapping lives in `extensions/qa-matrix/src/runners/contract/scenario-catalog.ts`.
7878

@@ -86,6 +86,7 @@ The full scenario id list is the `MatrixQaScenarioId` union in `extensions/qa-ma
8686
- media — `matrix-media-type-coverage`, `matrix-room-image-understanding-attachment`, `matrix-attachment-only-ignored`, `matrix-unsupported-media-safe`
8787
- routing — `matrix-room-autojoin-invite`, `matrix-secondary-room-*`
8888
- reactions — `matrix-reaction-*`
89+
- approvals — `matrix-approval-*` (exec/plugin metadata, chunked fallback, deny reactions, threads, and `target: "both"` routing)
8990
- restart and replay — `matrix-restart-*`, `matrix-stale-sync-replay-dedupe`, `matrix-room-membership-loss`, `matrix-homeserver-restart-resume`, `matrix-initial-catchup-then-incremental`
9091
- mention gating and allowlists — `matrix-mention-*`, `matrix-allowlist-*`, `matrix-multi-actor-ordering`, `matrix-inbound-edit-*`, `matrix-mxid-prefixed-command-block`, `matrix-observer-allowlist-override`
9192
- E2EE — `matrix-e2ee-*` (basic reply, thread follow-up, bootstrap, recovery key lifecycle, state-loss variants, server backup behavior, device hygiene, SAS / QR / DM verification, restart, artifact redaction)
@@ -112,7 +113,7 @@ Written to `--output-dir`:
112113

113114
- `matrix-qa-report.md` — Markdown protocol report (what passed, failed, was skipped, and why).
114115
- `matrix-qa-summary.json` — Structured summary suitable for CI parsing and dashboards.
115-
- `matrix-qa-observed-events.json` — Observed Matrix events from the driver and observer clients. Bodies are redacted unless `OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1`.
116+
- `matrix-qa-observed-events.json` — Observed Matrix events from the driver and observer clients. Bodies are redacted unless `OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1`; approval metadata is summarized with selected safe fields and truncated command preview.
116117
- `matrix-qa-output.log` — Combined stdout/stderr from the run. If `OPENCLAW_RUN_NODE_OUTPUT_LOG` is set, the outer launcher's log is reused instead.
117118

118119
The default output dir is `<repo>/.artifacts/qa-e2e/matrix-<timestamp>` so successive runs do not overwrite each other.

‎extensions/qa-matrix/src/runners/contract/runtime.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ type MatrixQaGatewayChild = {
4040
call(
4141
method: string,
4242
params: Record<string, unknown>,
43-
options?: { timeoutMs?: number },
43+
options?: { expectFinal?: boolean; timeoutMs?: number },
4444
): Promise<unknown>;
4545
restartAfterStateMutation?: (
4646
mutateState: (context: { stateDir: string }) => Promise<void>,
@@ -789,6 +789,8 @@ export async function runMatrixQaLive(params: {
789789
observerUserId: provisioning.observer.userId,
790790
gatewayRuntimeEnv: scenarioGateway.harness.gateway.runtimeEnv,
791791
gatewayStateDir: scenarioGateway.harness.gateway.runtimeEnv?.OPENCLAW_STATE_DIR,
792+
gatewayCall: async (method, params, opts) =>
793+
await scenarioGateway.harness.gateway.call(method, params ?? {}, opts),
792794
outputDir,
793795
registrationToken: harness.registrationToken,
794796
restartGateway: async () => {

‎extensions/qa-matrix/src/runners/contract/scenario-catalog.ts‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,12 @@ export type MatrixQaScenarioId =
4343
| "matrix-reaction-threaded"
4444
| "matrix-reaction-not-a-reply"
4545
| "matrix-reaction-redaction-observed"
46+
| "matrix-approval-exec-metadata-single-event"
47+
| "matrix-approval-exec-metadata-chunked"
48+
| "matrix-approval-plugin-metadata-single-event"
49+
| "matrix-approval-deny-reaction"
50+
| "matrix-approval-thread-target"
51+
| "matrix-approval-channel-target-both"
4652
| "matrix-restart-resume"
4753
| "matrix-post-restart-room-continue"
4854
| "matrix-initial-catchup-then-incremental"
@@ -266,6 +272,51 @@ const MATRIX_QA_E2EE_CLI_SETUP_CONFIG = {
266272
startupVerification: "off",
267273
} satisfies MatrixQaConfigOverrides;
268274

275+
const MATRIX_QA_APPROVAL_CHANNEL_CONFIG = {
276+
approvalForwarding: {
277+
exec: true,
278+
},
279+
dm: {
280+
enabled: true,
281+
},
282+
execApprovals: {
283+
enabled: true,
284+
target: "channel",
285+
},
286+
} satisfies MatrixQaConfigOverrides;
287+
288+
const MATRIX_QA_APPROVAL_CHUNKED_CONFIG = {
289+
...MATRIX_QA_APPROVAL_CHANNEL_CONFIG,
290+
chunkMode: "length",
291+
textChunkLimit: 280,
292+
} satisfies MatrixQaConfigOverrides;
293+
294+
const MATRIX_QA_APPROVAL_PLUGIN_CONFIG = {
295+
approvalForwarding: {
296+
plugin: true,
297+
},
298+
dm: {
299+
enabled: true,
300+
},
301+
execApprovals: {
302+
enabled: true,
303+
target: "channel",
304+
},
305+
} satisfies MatrixQaConfigOverrides;
306+
307+
const MATRIX_QA_APPROVAL_BOTH_CONFIG = {
308+
approvalForwarding: {
309+
exec: true,
310+
},
311+
dm: {
312+
enabled: true,
313+
},
314+
execApprovals: {
315+
enabled: true,
316+
target: "both",
317+
},
318+
} satisfies MatrixQaConfigOverrides;
319+
269320
export const MATRIX_QA_SCENARIOS: MatrixQaScenarioDefinition[] = [
270321
{
271322
id: "matrix-thread-follow-up",
@@ -518,6 +569,43 @@ export const MATRIX_QA_SCENARIOS: MatrixQaScenarioDefinition[] = [
518569
timeoutMs: 45_000,
519570
title: "Matrix reaction removals are observed as redactions",
520571
},
572+
{
573+
id: "matrix-approval-exec-metadata-single-event",
574+
timeoutMs: 75_000,
575+
title: "Matrix exec approval prompt carries structured metadata on one event",
576+
configOverrides: MATRIX_QA_APPROVAL_CHANNEL_CONFIG,
577+
},
578+
{
579+
id: "matrix-approval-exec-metadata-chunked",
580+
timeoutMs: 90_000,
581+
title: "Matrix exec approval prompt fallback keeps metadata on the first chunk",
582+
configOverrides: MATRIX_QA_APPROVAL_CHUNKED_CONFIG,
583+
},
584+
{
585+
id: "matrix-approval-plugin-metadata-single-event",
586+
timeoutMs: 75_000,
587+
title: "Matrix plugin approval prompt carries plugin metadata",
588+
configOverrides: MATRIX_QA_APPROVAL_PLUGIN_CONFIG,
589+
},
590+
{
591+
id: "matrix-approval-deny-reaction",
592+
timeoutMs: 75_000,
593+
title: "Matrix approval deny reaction resolves the metadata-bearing event",
594+
configOverrides: MATRIX_QA_APPROVAL_CHANNEL_CONFIG,
595+
},
596+
{
597+
id: "matrix-approval-thread-target",
598+
timeoutMs: 75_000,
599+
title: "Matrix approval prompt preserves thread targeting metadata",
600+
configOverrides: MATRIX_QA_APPROVAL_CHANNEL_CONFIG,
601+
},
602+
{
603+
id: "matrix-approval-channel-target-both",
604+
timeoutMs: 90_000,
605+
title: "Matrix approval target=both delivers channel and DM metadata once",
606+
topology: MATRIX_QA_DRIVER_DM_TOPOLOGY,
607+
configOverrides: MATRIX_QA_APPROVAL_BOTH_CONFIG,
608+
},
521609
{
522610
id: "matrix-restart-resume",
523611
standardId: "restart-resume",
@@ -985,6 +1073,8 @@ const MATRIX_QA_FAST_PROFILE_SCENARIO_IDS = [
9851073
"matrix-thread-isolation",
9861074
"matrix-top-level-reply-shape",
9871075
"matrix-reaction-notification",
1076+
"matrix-approval-exec-metadata-single-event",
1077+
"matrix-approval-exec-metadata-chunked",
9881078
"matrix-restart-resume",
9891079
"matrix-mention-gating",
9901080
"matrix-allowlist-block",

0 commit comments

Comments
 (0)