Summary

• Problem: When a Pi DAG node receives an SDK error result, the thrown error never reaches the executor's catch block because the async generator cleanup in bridgeSession hangs forever — session.dispose() closes Pi's I/O handles but doesn't settle session.prompt(). Bun drains its event loop and exits with code 0, leaving the workflow run stuck in status = 'running' indefinitely.
• Why it matters: Zombie runs block all subsequent runs on the same worktree path (path-lock) and require manual abandon. No automatic recovery exists.
• What changed: (C1) bridgeSession post-dispose await now races against a 10 s timeout so the event loop is never left with zero anchors; (C2) executeWorkflow gets a finally backstop that flips any still-running run to failed after all code paths complete; (C3) regression test added that intentionally passes a never-settling prompt() mock.
• What did NOT change: Claude and Codex providers, withIdleTimeout timing, DAG execution logic, process signal handling, and all existing behaviour on successful runs.

UX Journey

Before

User                    Archon CLI               Pi Provider
────                    ──────────               ───────────
archon workflow run ──▶ starts DAG run
                        (status = 'running')
                        streams to Pi ─────────▶ returns isError: true
                        dag-executor throws
                        [for-await calls .return()]
                        bridgeSession.finally:
                          session.dispose()
                          await promptPromise   ← hangs forever
                                                  (Bun event loop drains)
                        [process exits code 0]
                                                  catch block: NEVER RUNS
                                                  status stays 'running'
User sees:              $ (prompt returns, status = 'running' in DB)
archon workflow status  shows run stuck 'running' — zombie forever
archon workflow run ... "worktree is in use" error on next run

After

User                    Archon CLI               Pi Provider
────                    ──────────               ───────────
archon workflow run ──▶ starts DAG run
                        (status = 'running')
                        streams to Pi ─────────▶ returns isError: true
                        dag-executor throws
                        [for-await calls .return()]
                        bridgeSession.finally:
                          session.dispose()
                          [Promise.race: prompt vs 10s timeout]
                          timeout wins after 10s  ← event loop stays alive
                        throw propagates to catch
                        dag_node_failed logged
                        executeWorkflow catch runs
                        [finally backstop checks status]
                        status → 'failed'
User sees:              run status = 'failed' (not 'running')
archon workflow status  shows correct terminal state
archon workflow run ... next run proceeds normally

Architecture Diagram

Before

dag-executor.ts
  executeNodeInternal()
    for await (msg of withIdleTimeout(aiClient.sendQuery()))
      if (msg.isError) throw Error          ← throws
        [.return() called on withIdleTimeout]
          idle-timeout.ts finally:
            await generator.return(undefined)
              event-bridge.ts finally:
                session.dispose()           ← synchronous
                await promptPromise.catch() ← HANGS FOREVER (Pi bug)
                                              process exits, catch never runs

After

dag-executor.ts
  executeNodeInternal()
    for await (msg of withIdleTimeout(aiClient.sendQuery()))
      if (msg.isError) throw Error          ← throws
        [.return() called on withIdleTimeout]
          idle-timeout.ts finally:
            await generator.return(undefined)
              event-bridge.ts finally:      [~MODIFIED]
                session.dispose()
                [Promise.race([promptPromise, 10s timeout])]
                                            ← resolves within 10s
                throw propagates upward
  executor.ts executeWorkflow catch:
    failWorkflowRun(...)                    ← status = 'failed'
  executor.ts executeWorkflow finally:      [+NEW]
    if status still 'running': flip to 'failed'  ← backstop

Connection inventory:

Label Snapshot

• Risk: risk: low
• Size: size: S
• Scope: workflows, providers
• Module: workflows:executor, providers:pi

Change Metadata

• Change type: bug
• Primary scope: multi (workflows + providers)

Linked Issue

• Closes Workflow runs zombie in 'running' state when DAG node hits SDK error mid-stream #1561

Validation Evidence (required)

bun run validate

All checks passed:

• bun run check:bundled — PASS (bundled-defaults.generated.ts up to date, 36 commands, 20 workflows)
• bun run type-check — PASS (all 10 packages clean)
• bun run lint --max-warnings 0 — PASS (0 errors, 0 warnings)
• bun run format:check — PASS
• bun run test — PASS (full suite green; new bridgeSession timeout test passes in ~10.04s, dominated by the intentional 10s timeout it exercises)

The new regression test in event-bridge.test.ts directly validates the fix: a mock session whose prompt() returns a never-settling promise no longer hangs cleanup — the consumer's thrown error propagates within ~10 s. Without the fix, this test would hang indefinitely (caught by the 15 s test-level timeout).

Security Impact (required)

• New permissions/capabilities? No
• New external network calls? No
• Secrets/tokens handling changed? No
• File system access scope changed? No

Compatibility / Migration

• Backward compatible? Yes — existing Claude and Codex workflows are completely unaffected; the bridgeSession change is Pi-specific; the finally backstop in executor.ts is a no-op on runs that already finalize correctly
• Config/env changes? No
• Database migration needed? No

Human Verification (required)

• Verified scenarios: Full bun run validate suite passed (type-check, lint, format, tests, bundled-defaults check)
• Edge cases checked: (C1) getWorkflowRunStatus throw in backstop is caught and swallowed; (C2) failWorkflowRun throw in backstop is caught and logged; (C3) workflowRun undefined guard prevents backstop from running on run-creation failure; (C4) timeout fires after 10 s in the test, not sooner
• What was not verified: Live end-to-end run against a real Pi/Minimax provider (requires external API key)

Side Effects / Blast Radius (required)

• Affected subsystems/workflows: Pi provider cleanup path only; executor.ts finally backstop is universal but fires only when a run is still running post-execution (should be rare/never on non-Pi paths)
• Potential unintended effects: The 10 s post-dispose wait only affects already-errored Pi sessions; normal successful runs complete and close the generator cleanly before bridgeSession's finally is triggered for cleanup, so the race resolves immediately (prompt already settled)
• Guardrails: New executor.backstop_fail_failed log event emitted if the backstop's own failWorkflowRun call fails

Rollback Plan (required)

• Fast rollback command/path: Revert commit 853f491d (git revert 853f491d)
• Feature flags or config toggles: None
• Observable failure symptoms: Workflow runs stuck in running after a Pi SDK error; archon workflow status shows running indefinitely; subsequent runs on same worktree blocked with "worktree is in use"

Risks and Mitigations

• Risk: 10 s post-dispose timeout could delay test suite if many Pi tests use never-settling mocks
  • Mitigation: The single regression test is intentional and already accounts for this (~10 s). No other tests use never-settling prompt() mocks. Isolated in its own bun test invocation via the existing @archon/providers test batch configuration.

Summary by CodeRabbit

• Bug Fixes

  • Improved cleanup behavior to prevent blocking on unresolved operations.
  • Added safeguards to properly handle workflows that unexpectedly exit while still running, ensuring they are marked as failed and logged appropriately.

• Tests

  • Added regression tests validating cleanup resilience and workflow failure handling.