Agent Persona Exploration - 2026-04-04 #24431
Closed
Replies: 1 comment
-
|
This discussion has been marked as outdated by Agent Persona Explorer. A newer discussion is available at Discussion #24656. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
This report summarizes an exploration of how the agentic-workflows custom agent responds to workflow creation requests from 5 different software worker personas. Seven representative scenarios were tested and evaluated across 5 quality dimensions.
Persona Overview
Key Findings
safe-outputspull_request,workflow_run,schedule: weekly) are chosen with appropriate path filterstools.githubis used for GitHub API access (notnetwork.allowed), andplaywrightis correctly suggested for browser tasksTop Patterns
pull_request(path-filtered) for code automation,schedule: weekly(fuzzy) for digests,workflow_runfor reactive deployment monitoringtools.github(toolsets: default) in every scenario;bashfor local file analysis;playwrightfor visual tasks;network.allowedonly for external APIssafe-outputsfor writes, read-onlypermissions:, andnoopsafe-output for no-op paths — no insecure configurations observedView High Quality Responses (Score ≥ 4.6)
pm-1 — Weekly Feature Digest (5.0/5): Textbook minimal-footprint workflow. Pure
tools.githubread + onecreate-discussionsafe-output, no network access, fuzzy weekly schedule. Ideal reference example for PM-style reporting workflows.be-1 — DB Migration Risk Assessor (4.8/5): Excellent use of
paths: db/migrations/**trigger filter. Usesbash catfor local file analysis (more reliable than GitHub API fetches).hide-older-comments: trueon the comment safe-output keeps PR threads clean across re-pushes.be-2 — Performance Alert Triage (4.8/5): Notable insight:
roles: allis critical for monitoring-bot-created issues. Without it, the workflow silently skips the most important triggers. Strongtools.github defaultusage for commit/PR search.do-1 — Deployment Incident Creator (4.6/5): Correctly uses
workflow_runtrigger withnoopsafe-output for the success path. The deliberateclose-older-issues: falseon incident issues is a thoughtful safety practice — incidents should require manual triage.View Areas for Improvement (Score ≤ 4.2)
fe-1 — Visual Regression Reporter (4.0/5): The agent proposes a single workflow but a production-ready implementation requires two: one on
push to mainto capture baseline screenshots as artifacts, and one onpull_requestto fetch and compare. The agent does not proactively surface this architectural requirement. Prompt score suffered for insufficient guidance on baseline strategy.qa-1 — Coverage Regression (4.0/5): Artifact availability is timing-dependent — CI may still be running when the PR is opened. The agent proposes
pull_requesttrigger but doesn't suggest the superiorworkflow_runtrigger (keyed to CI completing), which guarantees artifacts exist. The multi-step chain (find run → download artifact → diff JSON → format comment) needs explicit prompt steps to avoid ambiguous LLM behavior.do-2 — Weekly Cost Digest (4.2/5): Secret binding gap — the agent correctly infers
COST_API_TOKENis needed but places it in the prompt description rather than providing guidance on secret declaration. Minor security gap: no HTTPS enforcement guidance for the internal API endpoint.Recommendations
Add artifact-workflow timing guidance to
.github/aw/create-agentic-workflow.md: When a user requests "read CI artifacts on PR open", proactively suggest theworkflow_runtrigger (keyed to the CI workflow completing) instead ofpull_requestto guarantee artifact availability. Add a heuristic: "If the task requires CI artifacts, useworkflow_runnotpull_request."Add multi-workflow composition guidance to
.github/aw/github-agentic-workflows.md: For visual regression and similar stateful comparisons, document the baseline + comparison two-workflow pattern explicitly. The agent should recognize when a single workflow cannot capture both sides of a diff (before/after state) and proactively explain the split.Add secret binding conventions to
.github/aw/create-agentic-workflow.md: Whennetwork.allowedis configured for an external API, the agent should remind users to declare the required secret (e.g.,API_TOKEN) as a GitHub repository secret and reference it in the prompt with a standard pattern. This prevents secret-handling ambiguity and reduces incomplete configurations.Scenario Score Breakdown
References: §23970400897
Beta Was this translation helpful? Give feedback.
All reactions