Skip to content

fix(scripts): use line-anchored regex to extract ARCHON_STATE_JSON markers#1695

Merged
Wirasm merged 3 commits into
devfrom
archon/task-archon-fix-github-issue-experimental-1778840374218
May 15, 2026
Merged

fix(scripts): use line-anchored regex to extract ARCHON_STATE_JSON markers#1695
Wirasm merged 3 commits into
devfrom
archon/task-archon-fix-github-issue-experimental-1778840374218

Conversation

@Wirasm

@Wirasm Wirasm commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: maintainer-standup-persist.ts matched ARCHON_STATE_JSON_BEGIN/END as arbitrary substrings, causing JSON.parse to fail when either marker appeared inside brief prose or as a substring within a JSON string value (e.g., a PR title referencing the bug fix itself).
  • Why it matters: The standup workflow's 20-minute synthesize node call is wasted and the run is reported as failed; the self-referential failure mode means any standup run that mentions this bug or PR fix(scripts): handle duplicate ARCHON_STATE_JSON_BEGIN blocks in persist #1676 will break.
  • What changed: Tier 1 delimiter extractor in .archon/scripts/maintainer-standup-persist.ts now uses line-anchored regex (^ARCHON_STATE_JSON_BEGIN$ / ^ARCHON_STATE_JSON_END$, m flag) with matchAll, selecting the last complete BEGIN/END pair. New test file .archon/scripts/maintainer-standup-persist.test.ts covers 7 cases (4 from PR fix(scripts): handle duplicate ARCHON_STATE_JSON_BEGIN blocks in persist #1676 + 3 new failure modes).
  • What did not change: Tier 2 JSON-wrapper fallback (Pi/Minimax path), synthesize node prompt, workflow YAML files, all packages/ code.

UX Journey

Before

standup workflow synthesize node (20 min)
  → stdout contains brief + ARCHON_STATE_JSON_BEGIN + json + ARCHON_STATE_JSON_END
  → in some runs: duplicate BEGIN, or marker string appears in prose/JSON value

persist script (Tier 1 extractor)
  → raw.indexOf('ARCHON_STATE_JSON_BEGIN')
    → finds first occurrence — may be inside prose ("PR #1676 — fix ARCHON_STATE_JSON_BEGIN blocks")
    → or last occurrence inside JSON value via lastIndexOf (PR #1676 partial fix)
  → raw.slice(beginIdx, endIdx) → invalid or truncated JSON
  → JSON.parse fails
  → script exits 1 ← standup run marked FAILED, 20-min work lost

After

persist script (Tier 1 extractor)
  → matchAll(/^ARCHON_STATE_JSON_BEGIN$/gm) — line-anchored, only whole-line matches
    → prose lines ("PR #1676 — fix ARCHON_STATE_JSON_BEGIN blocks") are longer than
      the marker alone; they do NOT match ^ARCHON_STATE_JSON_BEGIN$
    → JSON string values ("title": "...ARCHON_STATE_JSON_BEGIN...") are not whole lines
  → picks last END, last BEGIN before it → final complete pair
  → JSON.parse succeeds [✅]
  → script exits 0 — state written, brief stored

Architecture Diagram

Before

maintainer-standup-persist.ts (Tier 1)
  raw: string
  BEGIN = 'ARCHON_STATE_JSON_BEGIN'
  END   = 'ARCHON_STATE_JSON_END'
  beginIdx = raw.indexOf(BEGIN)   ← substring match, finds prose occurrences
  endIdx   = raw.indexOf(END)
  stateText = raw.slice(beginIdx + BEGIN.length, endIdx)  ← may be garbage
  JSON.parse(stateText)           ← throws

After

maintainer-standup-persist.ts (Tier 1) [~]
  BEGIN_RE = /^ARCHON_STATE_JSON_BEGIN$/gm  [+]
  END_RE   = /^ARCHON_STATE_JSON_END$/gm    [+]
  beginMatches = [...raw.matchAll(BEGIN_RE)] ← only whole-line matches  [+]
  endMatches   = [...raw.matchAll(END_RE)]                               [+]
  lastEnd   = endMatches[last]
  lastBegin = last beginMatch before lastEnd  ← final complete pair      [+]
  stateText = raw.slice(afterBeginIdx, lastEndIdx)  ← always valid JSON
  JSON.parse(stateText)           ← succeeds

Connection inventory:

From To Status Notes
maintainer-standup.yaml maintainer-standup-persist.ts unchanged script: node, stdin piped
maintainer-standup-minimax.yaml maintainer-standup-persist.ts unchanged same script, fix applies automatically
maintainer-standup-persist.ts state JSON file unchanged writes to $ARTIFACTS_DIR/state.json

Label Snapshot

  • Risk: risk: low
  • Size: size: S
  • Scope: scripts
  • Module: scripts:maintainer-standup-persist

Change Metadata

  • Change type: bug
  • Primary scope: multi (scripts + test)

Linked Issue

Validation Evidence (required)

bun run validate
# bundled-defaults: ✅ (36 commands, 20 workflows — up to date)
# bundled-skill:    ✅ (21 files — up to date)
# type-check:       ✅ all 10 packages green
# lint:             ✅ 0 errors, 0 warnings (--max-warnings 0)
# format:check:     ✅ all files clean
# tests:            ✅ all pass

bun test ./.archon/scripts/maintainer-standup-persist.test.ts
# 7 pass / 0 fail / 19 expect() calls

Test breakdown:

# Test Source
1 single BEGIN/END block succeeds PR #1676
2 duplicate BEGIN blocks — takes last complete block PR #1676
3 JSON-wrapper fallback works PR #1676
4 no valid format exits 1 PR #1676
5 marker substring in brief prose — not confused NEW
6 marker substring inside JSON string value — not confused NEW
7 duplicate BEGIN + marker in prose — last complete pair wins NEW

Test 6 fails on PR #1676's lastIndexOf approach, confirming the regression.

  • Evidence provided: All 7 tests pass; bun run validate clean.
  • No commands intentionally skipped.

Security Impact (required)

  • New permissions/capabilities? No
  • New external network calls? No
  • Secrets/tokens handling changed? No
  • File system access scope changed? No

Compatibility / Migration

  • Backward compatible? Yes — same stdin/stdout contract, same exit codes
  • Config/env changes? No
  • Database migration needed? No

Human Verification (required)

  • Verified scenarios: All 7 automated test cases pass, including the self-referential case (marker in prose) and the marker-in-JSON-value case.
  • Edge cases checked: No line-anchored BEGIN (Tier 2 fallback runs), multiple END markers (last used), BEGIN as first line (empty brief — acceptable).
  • What was not verified: Live standup run against real Claude output (requires live AI call); behaviour is fully covered by the test suite.

Side Effects / Blast Radius (required)

  • Affected subsystems/workflows: maintainer-standup.yaml and maintainer-standup-minimax.yaml both use the persist script — fix applies to both automatically.
  • Potential unintended effects: None anticipated. Line-anchored matching is strictly narrower than substring matching; previously-working inputs continue to match.
  • Guardrails: Stderr WARN emitted when more than one BEGIN marker is found (duplicate emission case), preserving PR fix(scripts): handle duplicate ARCHON_STATE_JSON_BEGIN blocks in persist #1676's diagnostic intent.

Rollback Plan (required)

  • Fast rollback: git revert 5d51f764 and re-push — script reverts to substring matching.
  • Feature flags: None.
  • Observable failure symptoms: persist script exits 1, workflow run status shows FAILED on the persist node.

Risks and Mitigations

  • Risk: matchAll not available in very old Bun versions.
    • Mitigation: String.prototype.matchAll is ES2020 (Bun 1.0+); all supported Archon targets meet this bar.

Summary by CodeRabbit

  • Bug Fixes

    • Improved error handling and logging for JSON parsing and file persistence failures.
    • Enhanced robustness in extracting JSON blocks from text input.
  • Tests

    • Added comprehensive test suite for maintainer state persistence.

Review Change Stack

The persist script extracted Claude's state-JSON block by substring-matching
the BEGIN/END markers, which gave a false match whenever the marker string
appeared inside the brief's prose or inside a JSON string value (e.g. a PR
title narrating this very bug). PR #1676's switch to lastIndexOf made the
self-referential case worse, since the last substring occurrence may now sit
inside a JSON value.

Match the markers only when they occupy an entire line (line-anchored
^...$ regex with the m flag) and pick the last END plus the last BEGIN
before it, so duplicate-emission and substring-in-content both resolve
correctly.

Changes:
- Replace indexOf/lastIndexOf substring matching in Tier 1 with
  line-anchored matchAll regex over BEGIN/END markers
- Add tests for marker substring in brief prose, marker substring inside
  state JSON value, and combined duplicate-BEGIN + marker-in-prose case
- Keep PR #1676's existing test cases (single block, duplicate BEGINs,
  JSON-wrapper fallback, no-valid-format exit 1)

Fixes #1674
@coderabbitai

coderabbitai Bot commented May 15, 2026

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 08ca7ef8-64ef-49c1-8438-c120f1eaf448

📥 Commits

Reviewing files that changed from the base of the PR and between e60af1d and 5d93944.

📒 Files selected for processing (2)
  • .archon/scripts/maintainer-standup-persist.test.ts
  • .archon/scripts/maintainer-standup-persist.ts

📝 Walkthrough

Walkthrough

Updated maintainer-standup-persist.ts to select the last complete ARCHON_STATE_JSON_BEGIN/ARCHON_STATE_JSON_END pair via line-anchored regex instead of indexOf, fixing failures when duplicate partial blocks are emitted. File writes are wrapped in try/catch with improved error logging. Added comprehensive test suite validating marker selection, failure modes, and regex robustness.

Changes

Maintainer standup persist robustness

Layer / File(s) Summary
Last-pair delimiter extraction with enhanced logging
.archon/scripts/maintainer-standup-persist.ts (lines 34–70)
Delimiter extraction rewritten from indexOf-based first-pair matching to regex-based line-anchored marker detection selecting the last complete pair. Warnings emitted for multiple BEGIN markers. Parse failures now log a JSON candidate preview. Handles restart scenarios where synthesize emits partial output followed by a complete block.
Persistence error handling and exit code management
.archon/scripts/maintainer-standup-persist.ts (lines 123–135)
Output path variables computed before try block. Directory creation and file writes wrapped in try/catch; on error logs PERSIST FAILED message and exits with code 1 instead of propagating unhandled exceptions.
Test infrastructure and comprehensive validation
.archon/scripts/maintainer-standup-persist.test.ts
runPersist helper spawns persist script in temp directory, captures stdio, parses metadata, reads persisted files, and cleans up. Tests validate single-block success, duplicate BEGIN handling (last-pair selection), JSON-wrapper fallback, marker-like substrings in prose and JSON string values, missing END (truncation error), no-marker failures (exit 1), and brief preamble stripping before first heading.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • coleam00/Archon#1480: Both PRs directly modify .archon/scripts/maintainer-standup-persist.ts's ARCHON_STATE_JSON_BEGIN/END marker parsing and state/brief persistence logic.

Poem

🐰 When synth blocks stuttered, our persist would break,
Chasing the first BEGIN (a parsing mistake!)—
Now regex finds the last complete pair,
Catches file errors with graceful care.
Tests guard against duplicate tears in the air! 🏃💨

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch archon/task-archon-fix-github-issue-experimental-1778840374218

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@Wirasm

Wirasm commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

Comprehensive PR Review

PR: #1695 — fix(scripts): use line-anchored regex to extract ARCHON_STATE_JSON markers
Reviewed by: 5 specialized agents (code-review, error-handling, test-coverage, comment-quality, docs-impact)
Date: 2026-05-15


Summary

The core fix is correct — replacing indexOf with line-anchored regex (/^ARCHON_STATE_JSON_BEGIN$/gm) properly prevents false matches when the marker string appears in PR titles or brief prose. The selection strategy (last END, last BEGIN before it) is algorithmically sound, and the test harness is robust. Two error-handling issues were found that should be addressed before marking ready.

Verdict: REQUEST_CHANGES

Severity Count
🔴 CRITICAL 0
🟠 HIGH 2
🟡 MEDIUM 3
🟢 LOW 4

🟠 High Issues (Should Fix Before Merge)

1. Silent fallthrough when all BEGIN markers follow the last END

📍 .archon/scripts/maintainer-standup-persist.ts:47-48

When beginMatches and endMatches are both non-empty the outer condition passes, but if every BEGIN position comes after the last END (structurally inverted markers), beginsBeforeEnd is empty and the inner guard silently skips to Tier 2 with no stderr output. The terminal error then says "neither delimiter nor JSON-wrapper matched" — actively misleading, because markers were present.

View fix
if (beginsBeforeEnd.length > 0) {
  // ... existing extraction code
} else {
  process.stderr.write(
    `WARN: ARCHON_STATE_JSON_BEGIN/END markers found but all BEGIN markers appear after the last END; skipping delimiter extraction.\n`,
  );
}

2. JSON parse error lacks candidate preview — message is not actionable

📍 .archon/scripts/maintainer-standup-persist.ts:63-67

The catch block logs err.message (e.g., Unexpected token < at position 0) but omits the stateText that failed. In a headless workflow the error must be self-contained — without the failed candidate the operator can't tell whether the model emitted truncated JSON, embedded HTML, etc.

View fix
} catch (err) {
  const preview = stateText.length > 200 ? stateText.slice(0, 200) + '…' : stateText;
  process.stderr.write(
    `Delimiter found but state JSON parse failed: ${(err as Error).message}\nFailed candidate (first 200 chars): ${preview}\n`,
  );
}

🟡 Medium Issues (Consider Fixing in Same Pass)

3. writeFileSync failures produce raw stack traces, not structured errors

📍 .archon/scripts/maintainer-standup-persist.ts:119,124,125

mkdirSync/writeFileSync are uncovered by try-catch — disk-full or permission errors produce a raw Bun stack trace instead of the script's clean PERSIST FAILED format.

View fix
try {
  mkdirSync(briefsDir, { recursive: true });
  writeFileSync(statePath, JSON.stringify(state, null, 2) + '\n');
  writeFileSync(briefPath, brief + '\n');
} catch (err) {
  process.stderr.write(
    `PERSIST FAILED: could not write output files: ${(err as Error).message}\n`,
  );
  process.exit(1);
}

4. No test for BEGIN present but END absent (truncated output)

📍 .archon/scripts/maintainer-standup-persist.test.ts (missing)

The most realistic truncation scenario (model cut off mid-generation after emitting BEGIN) is untested. The existing "no valid format exits 1" test uses input with no markers — a different code path.

Suggested test
test('BEGIN present but END absent — falls through to error', async () => {
  const input = [
    '# Standup',
    'ARCHON_STATE_JSON_BEGIN',
    '{"truncated": true',   // no END marker
  ].join('\n');
  const result = await runPersist(input);
  expect(result.exitCode).toBe(1);
  expect(result.stderr).toContain('PERSIST FAILED');
});

5. brief assignment comment describes mechanism, not the non-obvious WHY

📍 .archon/scripts/maintainer-standup-persist.ts:55

// Brief is everything before the first line-anchored BEGIN.
brief = raw.slice(0, beginMatches[0].index!).trim();

"First line-anchored BEGIN" describes the regex implementation but not why [0] (first marker) is used here while state extraction uses the last complete pair. That asymmetry is the part a reader needs explained.

Suggested: // Brief = everything before the first BEGIN; preserves prose intact even if state was emitted multiple times.


🟢 Low Issues

View 4 low-priority notes
Issue Location Note
CRLF line endings silently break $ matching ts:36-37 Theoretical — all known AI providers emit \n. YAGNI applies.
Test 6 description is misleading — case also passed under old indexOf code test.ts:100-121 Test is not harmful; consider relabeling or adding a clarifying comment.
Prose preamble stripping (lines 108-113) is pre-existing and untested ts:108-113 Low regression risk; not introduced by this PR.
Section header // ── Tier 1: ── explains what, not why ts:33 Pre-existing; kept for scannability alongside the Tier 2 header.

✅ What's Good

  • Selection strategy is correct — "last END, then last BEGIN before it" is the right heuristic for duplicate/truncated emissions and the strategy comment captures the WHY
  • beginMatches[0] for brief extraction is correct — preserves all human-readable prose even in duplicate-BEGIN scenarios
  • WARN fires only on successful parse — the duplicate-marker warning is inside the try block, avoiding confusing mixed signals on failure
  • Robust test harnessPromise.all([stdout, stderr]) prevents pipe-buffer deadlock; finally ensures temp dir cleanup on assertion failure
  • Scope discipline maintained — Tier 2 JSON-wrapper path untouched, no packages/ changes
  • All CLAUDE.md rules pass — type safety, no any, fail-fast, YAGNI, test determinism

No Docs Changes Required

Internal project script fix — transparent to users, no user-facing API or config change, no CHANGELOG entry needed.


Reviewed by Archon comprehensive-pr-review workflow

- Add WARN diagnostic when all BEGIN markers appear after last END
  (previously silent fallthrough produced misleading terminal error)
- Include 200-char candidate preview in JSON parse error message
  (error was unactionable in headless workflow without raw output)
- Wrap mkdirSync/writeFileSync in try-catch with structured PERSIST FAILED
  message instead of raw Bun stack trace on disk errors
- Fix brief assignment comment to explain WHY [0] vs last-pair asymmetry
- Add test: BEGIN present but END absent (truncated output) exits 1
- Add test: prose preamble before first heading is stripped from brief
- Add clarifying comment to Test 6 noting it is defence-in-depth, not
  a case that failed under the old indexOf approach
@Wirasm

Wirasm commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

⚡ Self-Fix Report (Aggressive)

Status: COMPLETE
Pushed: ✅ Changes pushed to archon/task-archon-fix-github-issue-experimental-1778840374218
Commit: 9c6ee53f
Philosophy: Fix everything unless clearly a new concern


Fixes Applied (5 total)

Severity Count
🔴 CRITICAL 0
🟠 HIGH 2
🟡 MEDIUM 2
🟢 LOW 1
View all fixes
  • Silent fallthrough when all BEGINs follow last END (maintainer-standup-persist.ts:47-48) — Added else branch with WARN diagnostic; previously fell silently to Tier 2 with a misleading terminal error
  • JSON parse error lacks candidate preview (maintainer-standup-persist.ts:63-67) — Added 200-char stateText preview to error message for actionable diagnosis in headless workflows
  • writeFileSync failures produce raw stack traces (maintainer-standup-persist.ts:119,124,125) — Wrapped mkdirSync/writeFileSync in try-catch with structured PERSIST FAILED prefix
  • brief assignment comment describes mechanism, not WHY (maintainer-standup-persist.ts:55) — Updated to: "Brief = everything before the first BEGIN; preserves prose intact even if state was emitted multiple times."
  • Test 6 description misleading (maintainer-standup-persist.test.ts:100) — Added clarifying comment noting it is defence-in-depth, not a case that failed under the old indexOf approach

Tests Added

  • BEGIN present but END absent (truncated output) — falls through to error — covers realistic context-truncation scenario (pairs with HIGH Issue 1 fix)
  • prose preamble before first heading is stripped from brief — covers pre-existing code path at lines 108-113 that was entirely untested

Skipped (2)

Finding Reason
CRLF line endings silently break $ matching YAGNI — all known AI providers emit \n; reviewer recommended Option A explicitly
// ── Tier 1: ── section header Pre-existing scaffolding; reviewer said no action needed

Suggested Follow-up Issues

(none — all actionable findings addressed)


Validation

✅ Type check | ✅ Lint | ✅ Tests (9 passed)


Self-fix by Archon · aggressive mode · fixes pushed to archon/task-archon-fix-github-issue-experimental-1778840374218

@Wirasm Wirasm marked this pull request as ready for review May 15, 2026 11:00
@Wirasm Wirasm merged commit 5d0225a into dev May 15, 2026
3 of 4 checks passed
@Wirasm Wirasm deleted the archon/task-archon-fix-github-issue-experimental-1778840374218 branch May 15, 2026 11:26
cropse pushed a commit to cropse/Archon that referenced this pull request May 19, 2026
…rkers (coleam00#1695)

* Fix: extract ARCHON_STATE_JSON markers as standalone lines (coleam00#1674)

The persist script extracted Claude's state-JSON block by substring-matching
the BEGIN/END markers, which gave a false match whenever the marker string
appeared inside the brief's prose or inside a JSON string value (e.g. a PR
title narrating this very bug). PR coleam00#1676's switch to lastIndexOf made the
self-referential case worse, since the last substring occurrence may now sit
inside a JSON value.

Match the markers only when they occupy an entire line (line-anchored
^...$ regex with the m flag) and pick the last END plus the last BEGIN
before it, so duplicate-emission and substring-in-content both resolve
correctly.

Changes:
- Replace indexOf/lastIndexOf substring matching in Tier 1 with
  line-anchored matchAll regex over BEGIN/END markers
- Add tests for marker substring in brief prose, marker substring inside
  state JSON value, and combined duplicate-BEGIN + marker-in-prose case
- Keep PR coleam00#1676's existing test cases (single block, duplicate BEGINs,
  JSON-wrapper fallback, no-valid-format exit 1)

Fixes coleam00#1674

* fix(scripts): address review findings in maintainer-standup-persist

- Add WARN diagnostic when all BEGIN markers appear after last END
  (previously silent fallthrough produced misleading terminal error)
- Include 200-char candidate preview in JSON parse error message
  (error was unactionable in headless workflow without raw output)
- Wrap mkdirSync/writeFileSync in try-catch with structured PERSIST FAILED
  message instead of raw Bun stack trace on disk errors
- Fix brief assignment comment to explain WHY [0] vs last-pair asymmetry
- Add test: BEGIN present but END absent (truncated output) exits 1
- Add test: prose preamble before first heading is stripped from brief
- Add clarifying comment to Test 6 noting it is defence-in-depth, not
  a case that failed under the old indexOf approach

* simplify: collapse multi-line comment blocks to single lines
@Wirasm Wirasm mentioned this pull request May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

maintainer-standup persist fails when synthesize emits duplicate ARCHON_STATE_JSON_BEGIN/END blocks

1 participant