You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Seventh run of the persona exploration series. Six scenarios tested across all five personas, targeting previously identified weak areas (CI failure trigger selection, external secrets handling). All scenarios simulated via general-purpose agents with gh-aw conventions — agentic-workflows agent type remains unavailable as a task tool.
Persona Overview
Scenarios Tested: 6 (targeting gaps from prior runs)
Trigger selection improved: workflow_run vs check_run question now definitively answered — workflow_run is correct for CI monitoring (fires once per workflow completion, not per job; payload carries run ID for full log access; avoids N duplicate events per run)
Secrets isolation pattern emerged: The credentials-health-check scenario produced a clean pattern where secrets live only in individual pre-step env: blocks, never in the agent context or frontmatter env: — agent receives only "valid" / "invalid" status strings
network: defaults: false legitimate: When the agent makes no direct outbound HTTP calls (all reads via GitHub MCP, writes via safe-outputs), setting defaults: false is valid and more restrictive
update-release confirmed real: The update_release safe-output exists with operation: replace|append|prepend — previously under-documented; repos (not releases) is the correct MCP toolset for release operations
Top Patterns (all 6 scenarios)
Pre-job data collection — heavy I/O (git checkouts, npm builds, API calls, log downloads) runs in a jobs: pre-step before the agent, which receives only structured summaries
Strict security defaults — strict: true (5/6), read-only permissions, all writes via safe-outputs, explicit bash allowlists
PR workflow hygiene — paths: filter + cancel-in-progress: true + update-existing: true on comments applied consistently to all 4 PR-triggered workflows
Explicit noop paths — every prompt names the exact noop condition with a ready-to-use message template
Anti-spam patterns — update-existing: true (PR comments), close-older-issues: true (recurring scheduled issues), dedup via issue search before create
High Quality Responses (all 6 scored 4.8/5)
CI Failure Analyzer (DevOps / workflow_run) — Score: 4.8
Dual-branch build solved cleanly in jobs: pre-step with rm -rf dist node_modules between branches
network: defaults: false — agent needs no outbound network
add-label with allowed: ["bundle-size-warning"] list for safety
Areas for Improvement
Trigger decision tree underdocumented — workflow_run vs check_run vs pull_request choice is non-obvious and recurs across runs. A concise decision matrix in .github/aw/create-agentic-workflow.md would reduce trial-and-error.
Secrets isolation pattern not yet a named pattern — The pre-step env-scoping approach for external APIs is excellent but ad-hoc. Naming it (e.g., "secrets firewall pattern") and documenting it in .github/aw/github-agentic-workflows.md would help users discover it.
strict: false trigger unclear — Agents sometimes set strict: false without fully understanding why. The current reason (inline shell in jobs: blocks) should be documented explicitly alongside what relaxations strict: false grants.
update-release discoverability — The PM scenario uncovered that update_release is a real safe-output but users asking "how do I update a release body?" won't find it easily. Adding an example to .github/aw/ docs would close this gap.
Recommendations
Add workflow_run trigger guide to .github/aw/create-agentic-workflow.md — include decision matrix (workflow_run vs check_run vs pull_request) with the key insight: workflow_run fires once per full workflow completion and provides run ID for log access
Document secrets isolation pattern in .github/aw/github-agentic-workflows.md — name the "pre-step secrets firewall" pattern: secrets in individual step env: blocks, agent receives only sanitized status outputs
Add update-release example to .github/aw/ documentation — it's a real safe-output that's underused; include operation: replace|append|prepend options and the repos toolset (not releases) for reading release data
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Seventh run of the persona exploration series. Six scenarios tested across all five personas, targeting previously identified weak areas (CI failure trigger selection, external secrets handling). All scenarios simulated via
general-purposeagents with gh-aw conventions —agentic-workflowsagent type remains unavailable as a task tool.Persona Overview
Key Findings
workflow_runvscheck_runquestion now definitively answered —workflow_runis correct for CI monitoring (fires once per workflow completion, not per job; payload carries run ID for full log access; avoids N duplicate events per run)env:blocks, never in the agent context or frontmatterenv:— agent receives only"valid"/"invalid"status stringsnetwork: defaults: falselegitimate: When the agent makes no direct outbound HTTP calls (all reads via GitHub MCP, writes via safe-outputs), settingdefaults: falseis valid and more restrictiveupdate-releaseconfirmed real: Theupdate_releasesafe-output exists withoperation: replace|append|prepend— previously under-documented;repos(notreleases) is the correct MCP toolset for release operationsTop Patterns (all 6 scenarios)
jobs:pre-step before the agent, which receives only structured summariesstrict: true(5/6), read-only permissions, all writes viasafe-outputs, explicit bash allowlistspaths:filter +cancel-in-progress: true+update-existing: trueon comments applied consistently to all 4 PR-triggered workflowsnoopcondition with a ready-to-use message templateupdate-existing: true(PR comments),close-older-issues: true(recurring scheduled issues), dedup via issue search before createHigh Quality Responses (all 6 scored 4.8/5)
CI Failure Analyzer (DevOps /
workflow_run) — Score: 4.8workflow_run+if: conclusion == 'failure'guard, explained why vscheck_runcache-memoryfor cross-run flakiness tracking; dedup search before creating issueCredentials Health Check (DevOps /
schedule) — Score: 4.8env:— strongest secrets isolation seen in any runif: needs.check-credentials.outputs.any-failed == 'true'skips agent entirely on clean runclose-older-issues: trueprevents stale "rotate credentials" issue accumulationOpenAPI Breaking Change Detector (Backend /
pull_request) — Score: 4.8sparse-checkoutfor openapi.yaml only — fast regardless of repo sizechange-summary5-value enum (breaking/clean/new_file/file_deleted/error) makes every branch explicitpersist-credentials: falseon both checkouts; oasdiff output capped at 50KBAccessibility Audit (QA /
pull_request+ Playwright) — Score: 4.8network.allowed)axe.run('#audit-root')prevents harness noise in resultsRelease Notes Generator (PM /
release: published) — Score: 4.8<!-- rng:generated -->marker as idempotency guard prevents overwriting manual editsrepostoolset (notreleases) forlist_releases/get_release_by_tagoperation: replacevsprepend— correctly chose replace for full body ownershipBundle Size Monitor (Frontend /
pull_request+ dual-branch build) — Score: 4.8jobs:pre-step withrm -rf dist node_modulesbetween branchesnetwork: defaults: false— agent needs no outbound networkadd-labelwithallowed: ["bundle-size-warning"]list for safetyAreas for Improvement
Trigger decision tree underdocumented —
workflow_runvscheck_runvspull_requestchoice is non-obvious and recurs across runs. A concise decision matrix in.github/aw/create-agentic-workflow.mdwould reduce trial-and-error.Secrets isolation pattern not yet a named pattern — The pre-step env-scoping approach for external APIs is excellent but ad-hoc. Naming it (e.g., "secrets firewall pattern") and documenting it in
.github/aw/github-agentic-workflows.mdwould help users discover it.strict: falsetrigger unclear — Agents sometimes setstrict: falsewithout fully understanding why. The current reason (inline shell injobs:blocks) should be documented explicitly alongside what relaxationsstrict: falsegrants.update-releasediscoverability — The PM scenario uncovered thatupdate_releaseis a real safe-output but users asking "how do I update a release body?" won't find it easily. Adding an example to.github/aw/docs would close this gap.Recommendations
workflow_runtrigger guide to.github/aw/create-agentic-workflow.md— include decision matrix (workflow_run vs check_run vs pull_request) with the key insight:workflow_runfires once per full workflow completion and provides run ID for log access.github/aw/github-agentic-workflows.md— name the "pre-step secrets firewall" pattern: secrets in individual stepenv:blocks, agent receives only sanitized status outputsupdate-releaseexample to.github/aw/documentation — it's a real safe-output that's underused; includeoperation: replace|append|prependoptions and therepostoolset (notreleases) for reading release dataReferences:
Beta Was this translation helpful? Give feedback.
All reactions