Phase 1 & 2: Squad workflows + failure detection infrastructure #322#324
Merged
spboyer merged 10 commits intoJun 15, 2026
Merged
Conversation
Replace TODO placeholders in squad CI/preview/release workflows with: - Go mod verify, go vet, go test with coverage, race detector - Node.js web UI build (Node 22 with npm cache) - Binary build and integration test for end-to-end validation - Release creation with gh CLI (main) and pre-release (insider) All workflows now mirror go-ci.yml proven build/test pattern: - squad-ci.yml: Runs on dev/preview/main/insider branches and PRs - squad-preview.yml: Validates on preview branch pushes - squad-release.yml: Runs tests and creates release on main - squad-insider-release.yml: Runs tests and creates pre-release on insider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5 tasks
2b678a7 to
e552ada
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to move the repository’s Squad automation from “stub” workflows toward real, enforceable CI/triage/release mechanics for a Go-based project, and adds/updates a large set of Squad templates and operational references to support that.
Changes:
- Adds many new
.squad/templates/*references (worktrees, wiring guides, state/notes protocols, Ralph/Rai/Scribe templates, etc.) and updates Squad-related configuration. - Updates GitHub triage/labeling workflows to slugify member names consistently for
squad:{member}labels. - Introduces a Node-based Ralph triage script intended to run in
squad-heartbeat.yml.
Show a summary per file
| File | Description |
|---|---|
| web/dist/index.html | Updates dashboard asset references |
| .squad/templates/worktree-reference.md | Adds worktree/root-resolution guidance |
| .squad/templates/workflow-wiring-guide.md | Adds workflow wiring/enforcement guide |
| .squad/templates/workflow-wiring-appendix-b-documenter.md | Adds documenter follow-up walkthrough |
| .squad/templates/workflow-wiring-appendix-a-code-reviewer.md | Adds reviewer gate walkthrough |
| .squad/templates/spawn-reference.md | Adds spawn prompt/reference template |
| .squad/templates/skill.md | Adds SKILL.md template |
| .squad/templates/session-init-reference.md | Adds session init/update-check reference |
| .squad/templates/scripts/notes/write-note.ps1 | Adds git-notes write helper script |
| .squad/templates/scripts/notes/fetch.ps1 | Adds git-notes fetch/merge helper script |
| .squad/templates/scribe-charter.md | Adds Scribe charter template |
| .squad/templates/run-output.md | Adds multi-agent run output template |
| .squad/templates/routing.md | Adds routing template |
| .squad/templates/roster.md | Adds roster template |
| .squad/templates/raw-agent-output.md | Adds raw-output appendix template |
| .squad/templates/ralph-triage.js | Adds Ralph auto-triage script for Actions |
| .squad/templates/ralph-reference.md | Adds Ralph monitoring reference |
| .squad/templates/ralph-circuit-breaker.md | Adds model rate-limit fallback template |
| .squad/templates/rai-policy.md | Adds RAI policy template |
| .squad/templates/Rai-charter.md | Adds Rai charter template |
| .squad/templates/prd-intake.md | Adds PRD intake template |
| .squad/templates/plugin-marketplace.md | Adds plugin marketplace guidance |
| .squad/templates/package.json | Sets template package.json to commonjs |
| .squad/templates/orchestration-log.md | Adds orchestration log entry template |
| .squad/templates/notes-protocol.md | Adds git-notes protocol/contract doc |
| .squad/templates/multi-agent-format.md | Adds multi-agent artifact format guidance |
| .squad/templates/model-selection-reference.md | Adds model selection reference |
| .squad/templates/mcp-config.md | Adds MCP config guidance/examples |
| .squad/templates/machine-capabilities.md | Adds machine capability routing template |
| .squad/templates/loop.md | Adds squad loop template |
| .squad/templates/keda-scaler.md | Adds KEDA scaler concept doc/template |
| .squad/templates/issue-lifecycle.md | Adds issue→branch→PR→merge lifecycle template |
| .squad/templates/identity/wisdom.md | Adds wisdom template |
| .squad/templates/identity/now.md | Adds “now” focus template |
| .squad/templates/history.md | Adds project history template |
| .squad/templates/fact-checker-charter.md | Adds fact-checker charter template |
| .squad/templates/copilot-instructions.md | Adds copilot instructions template |
| .squad/templates/copilot-agent.md | Adds @copilot roster/reference template |
| .squad/templates/cooperative-rate-limiting.md | Adds cooperative rate limiting template |
| .squad/templates/constraint-tracking.md | Adds constraint budget tracking template |
| .squad/templates/client-compatibility-reference.md | Adds client compatibility reference |
| .squad/templates/charter.md | Adds generic agent charter template |
| .squad/templates/ceremony-reference.md | Adds ceremony configuration reference |
| .squad/templates/ceremonies.md | Adds ceremonies template |
| .squad/templates/casting/Futurama.json | Adds Futurama casting universe list |
| .squad/templates/casting-registry.json | Adds casting registry template |
| .squad/templates/casting-reference.md | Adds casting system reference |
| .squad/templates/casting-policy.json | Adds casting policy template (incl. Futurama) |
| .squad/templates/casting-history.json | Adds casting history template |
| .squad/templates/after-agent-reference.md | Adds “after agent work” reference |
| .squad/memory/index.json | Adds/initializes memory index |
| .squad/memory/config.json | Adds memory config |
| .squad/memory/audit.jsonl | Adds memory audit log file |
| .squad/casting-policy.json | Extends installed casting policy with Futurama |
| .mcp.json | Adds MCP server config for squad_state |
| .gitignore | Ignores additional .squad/* state/cache paths |
| .github/workflows/sync-squad-labels.yml | Slugifies squad label names; label sync updates |
| .github/workflows/squad-triage.yml | Slugifies assigned squad labels during triage |
| .github/workflows/squad-heartbeat.yml | Removes commented cron; runs Ralph triage script |
| .github/copilot-instructions.md | Adds coordinator canary check instructions |
| .copilot/skills/test-discipline/SKILL.md | Adds test-discipline skill |
| .copilot/skills/squad-version-check/SKILL.md | Adds squad version check skill |
| .copilot/skills/squad-conventions/SKILL.md | Adds squad conventions skill |
| .copilot/skills/squad-commands/SKILL.md | Adds squad commands catalog skill |
| .copilot/skills/session-recovery/SKILL.md | Adds session recovery skill |
| .copilot/skills/secret-handling/SKILL.md | Adds secret-handling skill |
| .copilot/skills/reviewer-protocol/SKILL.md | Adds reviewer protocol skill |
| .copilot/skills/git-workflow/SKILL.md | Adds git workflow skill |
| .copilot/skills/error-recovery/SKILL.md | Adds error recovery skill |
| .copilot/skills/agent-collaboration/SKILL.md | Adds agent collaboration skill |
Copilot's findings
- Files reviewed: 6/6 changed files
- Comments generated: 0
e552ada to
547e93a
Compare
547e93a to
d498660
Compare
d498660 to
da08d3e
Compare
Phase 2: Implement genuine failure-handling for evaluation runs - Add failures/handler.go with CaptureFailure method - Capture stderr/stdout (10KB truncation), exit code, failed validators - Extract error patterns via regex (timeout, OOM, permission, connection, panic, exception) - Generate human-readable triage summary in Markdown with actionable recommendations - Extend RunResult.FailureArtifacts for diagnostic data in results JSON - Add comprehensive test coverage (handler_test.go) - All tests passing, 79.1% coverage maintained Next: Wire handler into EvalRunner, add --auto-file-issue flag, display triage in CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
da08d3e to
034c7a0
Compare
#322 - Revert go.mod back to 1.26 (dependency azure-dev/cli/azd requires >= 1.26) - Restore go-version: 1.26 in go-ci.yml and squad-ci.yml - Add linters.exclusions.paths to .golangci.yml to skip web/node_modules and web/dist (uses v2 schema: linters.exclusions.paths) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…322 v4 with 'latest' resolves to v1.64.8 (built with Go 1.24) which cannot lint Go 1.26 projects. Upgrade to action@v7 with pinned v2.10.1 (built with Go 1.26) to match go-ci.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add exitCode parameter to CaptureFailure and populate ExitCode field - Sort FailedGraders for deterministic output - Sort extractErrorPatterns result for deterministic output - Fix contains() helper to use strings.Contains - Fix TestTruncate: correct wantLen and assert actual length - Fix TestExtractErrorPatterns: assert specific expected patterns - Update TestCaptureFailure: pass exit code, assert ExitCode field - Drop unused contains() helper (now using strings.Contains directly) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use 'gh release view' to check if a release exists before creating, so genuine failures (permission errors, network issues) surface as workflow failures rather than being swallowed by '|| echo'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- handler.go: use time.Now().UTC() for cross-machine consistency - handler.go: filter empty strings from extractErrorPatterns matches - handler.go: fix truncate to enforce hard maxLen (prefix len = maxLen - suffix len) - handler_test.go: update TestTruncate wantLen to maxLen (30) after hard-limit fix - squad-preview/release/insider-release.yml: add favicon.svg to web/dist stub - squad-release/insider-release.yml: change 'passed' to 'completed' in integration test message since exit code 1 (eval failures) is treated as non-fatal - go-ci.yml lint job: fix go-version indentation to 10 spaces under with: - squad-ci.yml lint job: fix go-version indentation to 10 spaces under with: Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- handler_test.go: remove unused truncSuffix const (fixes lint) - handler.go: use regexp.MustCompile for hard-coded patterns to surface typos at runtime instead of silently ignoring compile errors - squad-ci.yml: add lfs: true to lint job checkout to match build job - .golangci.yml exclusions.paths is valid v2 config (config verify passes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In golangci-lint v2, path exclusions belong under the top-level exclusions key, not under linters.exclusions. Moving them ensures they are applied correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 15, 2026
…326) * feat(phase1): Implement real Go build/test steps in squad workflows #322 Replace TODO placeholders in squad CI/preview/release workflows with: - Go mod verify, go vet, go test with coverage, race detector - Node.js web UI build (Node 22 with npm cache) - Binary build and integration test for end-to-end validation - Release creation with gh CLI (main) and pre-release (insider) All workflows now mirror go-ci.yml proven build/test pattern: - squad-ci.yml: Runs on dev/preview/main/insider branches and PRs - squad-preview.yml: Validates on preview branch pushes - squad-release.yml: Runs tests and creates release on main - squad-insider-release.yml: Runs tests and creates pre-release on insider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add failure detection and artifact capture infrastructure #322 Phase 2: Implement genuine failure-handling for evaluation runs - Add failures/handler.go with CaptureFailure method - Capture stderr/stdout (10KB truncation), exit code, failed validators - Extract error patterns via regex (timeout, OOM, permission, connection, panic, exception) - Generate human-readable triage summary in Markdown with actionable recommendations - Extend RunResult.FailureArtifacts for diagnostic data in results JSON - Add comprehensive test coverage (handler_test.go) - All tests passing, 79.1% coverage maintained Next: Wire handler into EvalRunner, add --auto-file-issue flag, display triage in CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): restore Go 1.26, fix golangci-lint to exclude web/node_modules #322 - Revert go.mod back to 1.26 (dependency azure-dev/cli/azd requires >= 1.26) - Restore go-version: 1.26 in go-ci.yml and squad-ci.yml - Add linters.exclusions.paths to .golangci.yml to skip web/node_modules and web/dist (uses v2 schema: linters.exclusions.paths) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): update Squad CI lint to use golangci-lint-action@v7 v2.10.1 #322 v4 with 'latest' resolves to v1.64.8 (built with Go 1.24) which cannot lint Go 1.26 projects. Upgrade to action@v7 with pinned v2.10.1 (built with Go 1.26) to match go-ci.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address review comments in failure handler - Add exitCode parameter to CaptureFailure and populate ExitCode field - Sort FailedGraders for deterministic output - Sort extractErrorPatterns result for deterministic output - Fix contains() helper to use strings.Contains - Fix TestTruncate: correct wantLen and assert actual length - Fix TestExtractErrorPatterns: assert specific expected patterns - Update TestCaptureFailure: pass exit code, assert ExitCode field - Drop unused contains() helper (now using strings.Contains directly) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: replace error-suppressing release creation with idempotent check Use 'gh release view' to check if a release exists before creating, so genuine failures (permission errors, network issues) surface as workflow failures rather than being swallowed by '|| echo'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address all 14 new review comments from PR #324 - handler.go: use time.Now().UTC() for cross-machine consistency - handler.go: filter empty strings from extractErrorPatterns matches - handler.go: fix truncate to enforce hard maxLen (prefix len = maxLen - suffix len) - handler_test.go: update TestTruncate wantLen to maxLen (30) after hard-limit fix - squad-preview/release/insider-release.yml: add favicon.svg to web/dist stub - squad-release/insider-release.yml: change 'passed' to 'completed' in integration test message since exit code 1 (eval failures) is treated as non-fatal - go-ci.yml lint job: fix go-version indentation to 10 spaces under with: - squad-ci.yml lint job: fix go-version indentation to 10 spaces under with: Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address 3 new lint/review issues after second review pass - handler_test.go: remove unused truncSuffix const (fixes lint) - handler.go: use regexp.MustCompile for hard-coded patterns to surface typos at runtime instead of silently ignoring compile errors - squad-ci.yml: add lfs: true to lint job checkout to match build job - .golangci.yml exclusions.paths is valid v2 config (config verify passes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: move exclusions.paths to top-level in golangci.yml v2 config In golangci-lint v2, path exclusions belong under the top-level exclusions key, not under linters.exclusions. Moving them ensures they are applied correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: complete issue 322 reliability loops and triage automation #322 - add CLI auto issue filing and triage highlights for failed runs - wire failure artifact capture into runner execution paths - add weekly regression loop workflow with baseline artifact comparison and issue upsert - harden auto-merge workflow with trusted-author + label gates - document new run flag and CI workflows, and fix lint package resolution Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove invalid exclusions from golangci.yml golangci-lint config verify was failing because the top-level 'exclusions' property is not valid in the current schema. Removed this section as the web directory is not a Go package and does not need explicit exclusion. Fixes: Go Build and Test CI failure due to golangci-lint config verification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Phase 1 (real squad workflows) and Phase 2 (genuine failure-handling) of the continuous delivery pipeline for issue #322.
Phase 1: Replace Squad Workflow Stubs with Real Build & Test Steps
Updated all squad workflow files to replace TODO placeholders with actual Go build/test commands:
Files Modified
.github/workflows/squad-ci.yml.github/workflows/squad-preview.yml.github/workflows/squad-release.ymlgh release creategit describe --tags --always--latestflag for main branch releases.github/workflows/squad-insider-release.yml--prereleaseflagTest Status: All workflow syntax valid, Go build/test patterns proven via existing go-ci.yml
Phase 2: Add Genuine Failure-Handling for Eval Runs
Implements structured failure detection and remediation in the eval pipeline.
Files Created
internal/failures/handler.go— Failure detection moduleCaptureFailure()captures stderr/stdout (10KB truncation), exit code, failed validatorsextractErrorPatterns()identifies common error patterns via regex:generateTriageSummary()produces human-readable Markdown summary with:generateRecommendations()suggests fixes based on detected patternsinternal/failures/handler_test.go— Comprehensive test coverageTestCaptureFailure: Verifies artifact capture, grader trackingTestExtractErrorPatterns: Tests pattern matchingTestGenerateTriageSummary: Validates summary structureTestTruncate: Verifies size limitsFiles Modified
internal/models/outcome.go(prior turn)FailureArtifactsstruct toRunResultHow It Works
When an evaluation run fails:
RunResult.FailureArtifactsfor inclusion in results.jsonTest Results
Relates To
Phases 1 & 2 of Issue #322 continuous delivery pipeline
Next Steps (Phase 2 continuation)
EvalRunner.Run()to auto-capture failures--auto-file-issuefor optional GitHub issue creationrunner_test.gosite/src/content/docs/guides/Auto-merge: This PR will auto-merge once all required status checks pass.