fix(yolo-agents): require claim verification before reporting fires#202
fix(yolo-agents): require claim verification before reporting fires#202auroracapital merged 1 commit intomainfrom
Conversation
The 2026-05-01 YOLO session produced multiple false-positive fires that wasted orchestrator/operator time: - CTO claimed `stagery-api` prd was missing `CLERK_AUTHORIZED_PARTIES` — the var was set; only `CORS_ORIGINS` was empty. - CTO/COO flagged `healify-langgraphs-prod` desired=0 as "may still be 95% traffic path" — verified intentional Phase 19.4 Stage 2 decomm (commit `7ada4026 chore(decomm): disable prod deploy workflow`). - COO referenced a `stg` Doppler config that does not exist on stagery-api (configs are `dev`, `dev_personal`, `dev_local`, `prd`). Root cause: the agent prompts told them to be "brutally honest" but did not require them to verify external state before asserting it. Stale docs and pattern-matching from STATE.md notes were promoted to P0 claims. Add a CLAIM VERIFICATION GUARDRAIL section to all four C-suite agents (yolo-ceo, yolo-cto, yolo-cfo, yolo-coo). Same block in each — pulls agents toward concrete verification commands (doppler secrets get, git log --grep decomm) and forbids common false-positive output patterns. Anything that can't be verified must be labeled `UNVERIFIED`. Placement: before the existing DESTRUCTIVE ACTION GUARDRAIL block, so both pre-action checks live next to each other at the bottom of each agent file.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThis PR adds Claim Verification Guardrails to four YOLO C-suite agent instruction files (CEO, CFO, COO, CTO) mandating ground-truth verification before asserting external-state issues such as missing secrets, broken services, wrong config values, or abandoned projects. The COO file also extends its Destructive Action Guardrail with stricter project-status verification and confirmation requirements. ChangesUnified Claim Verification Guardrail Initiative
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
…203) Patch release rolling up the two YOLO reliability fixes merged this evening: - PR #201: extract STATE.md walker to bin/ops-gsd-states for shell portability. Restores Phase 1 data-gathering after the inline shell block silently aborted under sh/dash on the bash-only `${d/#\~/$HOME}` expansion. - PR #202: add CLAIM VERIFICATION GUARDRAIL to yolo-ceo / yolo-cto / yolo-cfo / yolo-coo so the agents stop reporting false-positive fires (verify against doppler/aws/git log before asserting; label UNVERIFIED when verification is impossible). Bumps: - plugin.json: 2.1.0 -> 2.1.1 - marketplace.json: 2.1.0 -> 2.1.1 - package.json (bin): 1.7.0 -> 1.7.1 CHANGELOG entry under [2.1.1] with details on both fixes and the real-world false-positive cases that motivated the verification rule. No breaking changes; drop-in replacement for 2.1.0.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.
Reviewed by Cursor Bugbot for commit 86d23b9. Configure here.
| **Mandatory verification rules:** | ||
|
|
||
| 1. **"Secret X is missing" — verify with the secret store.** | ||
| Doppler: `doppler secrets get <NAME> --project <p> --config <c> --plain` (returns empty if unset, the value if set). Never claim a secret is missing without this check. If unsure which configs exist, run `doppler configs --project <p>` first — do not assume names like `stg` / `prd` / `prod` exist. |
There was a problem hiding this comment.
Incorrect Doppler CLI behavior described contradicts rule 3
High Severity
Rule 1 states doppler secrets get ... --plain "(returns empty if unset, the value if set)" — this is factually wrong and directly contradicts rule 3 ("An empty string is different from an unset key — distinguish them"). The Doppler CLI actually returns an error for a nonexistent secret and an empty string for a secret set to empty. Agents following rule 1's parenthetical will still classify empty-valued secrets as "missing," which is exactly the false-positive pattern this PR exists to fix.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 86d23b9. Configure here.
| 1. **"Secret X is missing" — verify with the secret store.** | ||
| Doppler: `doppler secrets get <NAME> --project <p> --config <c> --plain` (returns empty if unset, the value if set). Never claim a secret is missing without this check. If unsure which configs exist, run `doppler configs --project <p>` first — do not assume names like `stg` / `prd` / `prod` exist. | ||
| AWS Secrets Manager: `aws secretsmanager get-secret-value --secret-id <arn>`. | ||
| GitHub Actions: `gh secret list --repo <r>`. |
There was a problem hiding this comment.
Bug: The documentation for doppler secrets get incorrectly states it returns an empty string for unset secrets, when it actually errors. This can mislead agent behavior.
Severity: MEDIUM
Suggested Fix
Update the documentation to accurately reflect the behavior of doppler secrets get. Clarify that it errors by default for an unset secret and that an empty string is returned only when a secret is explicitly set to an empty value. Mention the --no-exit-on-missing-secret flag as the way to alter the default erroring behavior.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: claude-ops/agents/yolo-ceo.md#L104-L107
Potential issue: The documentation for the `doppler secrets get ... --plain` command
incorrectly states that it 'returns empty if unset'. In reality, the Doppler CLI errors
and exits with a non-zero status code for an unset secret. An empty string is returned
only when a secret is explicitly set to an empty value. This misinformation can mislead
an agent into conflating a missing secret with one that is intentionally empty,
potentially causing it to handle secret retrieval incorrectly and bypass intended
safeguards.
Did we get this right? 👍 / 👎 to inform future reviews.
Summary
The 2026-05-01 YOLO session produced multiple false-positive fires:
stagery-apiprd was missingCLERK_AUTHORIZED_PARTIES— the var was set; onlyCORS_ORIGINSwas empty.healify-langgraphs-proddesired=0 as a fire — verified intentional Phase 19.4 Stage 2 decomm (commit7ada4026 chore(decomm): disable prod deploy workflow).stgconfig that does not exist onstagery-api(configs aredev,dev_personal,dev_local,prd).Root cause: the C-suite prompts told agents to be "brutally honest" but did not require them to verify external state before asserting it. Stale STATE.md / planning notes were promoted to P0 claims.
Change
Add a
## CLAIM VERIFICATION GUARDRAILsection to all four agents (yolo-ceo,yolo-cto,yolo-cfo,yolo-coo). Identical block in each, placed before the existingDESTRUCTIVE ACTION GUARDRAILso both pre-action checks live together.The block:
doppler secrets get,aws secretsmanager get-secret-value,git log --grep decomm)UNVERIFIEDlabeling when verification is impossible (rate limit, missing creds)Test plan
## DESTRUCTIVE ACTION GUARDRAIL(pre-existing) in each file/ops:ops-yolorun produces noMISSINGclaims for vars that are actually set, and noBROKENclaims for services with recent intentional decomm commitsNote
Low Risk
Low risk: prompt-only documentation changes that tighten agent output requirements without modifying runtime code paths.
Overview
Adds a new
## CLAIM VERIFICATION GUARDRAILsection toyolo-ceo,yolo-cto,yolo-cfo, andyolo-cooagent prompt files, inserted before the existing destructive-action guardrails.The guardrail requires agents to verify claims about missing secrets, broken services, wrong config values, or abandoned projects using concrete commands (e.g.,
doppler secrets get,aws secretsmanager get-secret-value, repogit logchecks), and to label anything they can’t confirm asUNVERIFIEDwhile banning common false-positive “fire” phrasing without ground-truth reads.Reviewed by Cursor Bugbot for commit 86d23b9. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit