Skip to content

Latest commit

 

History

History
186 lines (173 loc) · 9.77 KB

File metadata and controls

186 lines (173 loc) · 9.77 KB
description Smoke Copilot
true
schedule workflow_dispatch label_command reaction status-comment github-token
every 12h
name events
smoke
pull_request
eyes
true
${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}
permissions
contents pull-requests issues discussions actions
read
read
read
read
read
name Smoke Copilot
engine
id max-continuations
copilot
2
imports
shared/github-guard-policy.md
shared/gh.md
shared/reporting.md
shared/github-queries-mcp-script.md
shared/mcp/serena-go.md
shared/observability-otlp.md
network
allowed
defaults
node
github
playwright
tools
agentic-workflows cache-memory edit bash github playwright web-fetch
true
*
min-integrity trusted-users
approved
pelikhan
runtimes
go
version
1.25
safe-outputs
allowed-domains upload-artifact add-comment create-issue create-discussion create-pull-request-review-comment submit-pull-request-review reply-to-pull-request-review-comment add-labels remove-labels set-issue-type dispatch-workflow jobs messages
default-safe-outputs
max-uploads default-retention-days max-retention-days allow
1
1
1
skip-archive
true
allowed-repos hide-older-comments max
github/gh-aw
true
2
expires group close-older-issues close-older-key labels
2h
true
true
smoke-copilot
automation
testing
category labels expires close-older-discussions close-older-key max
announcements
ai-generated
2h
true
smoke-copilot
1
max
5
max
5
allowed allowed-repos
smoke-copilot
github/gh-aw
allowed
smoke
workflows max
haiku-printer
1
send-slack-message
description runs-on output inputs permissions steps
Send a message to Slack (stub for testing)
ubuntu-latest
Slack message stub executed!
message
description required default type
The message to send
false
string
contents
read
name run
Stub Slack message
echo "🎭 This is a stub - not sending to Slack" if [ -f "$GH_AW_AGENT_OUTPUT" ]; then MESSAGE=$(cat "$GH_AW_AGENT_OUTPUT" | jq -r '.items[] | select(.type == "send_slack_message") | .message') echo "Would send to Slack: $MESSAGE" { echo "### 📨 Slack Message Stub" echo "**Message:** $MESSAGE" echo "" echo "> ℹ️ This is a stub for testing purposes. No actual Slack message is sent." } >> "$GITHUB_STEP_SUMMARY" else echo "No agent output found" fi
append-only-comments footer run-started run-success run-failure
true
> 📰 *BREAKING: Report filed by [{workflow_name}]({run_url})*{effective_tokens_suffix}{history_link}
📰 BREAKING: [{workflow_name}]({run_url}) is now investigating this {event_type}. Sources say the story is developing...
📰 VERDICT: [{workflow_name}]({run_url}) has concluded. All systems operational. This is a developing story. 🎤
📰 DEVELOPING STORY: [{workflow_name}]({run_url}) reports {status}. Our correspondents are investigating the incident...
timeout-minutes 15
strict false

Smoke Test: Copilot Engine Validation

IMPORTANT: Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.

Test Requirements

  1. GitHub MCP Testing: Review the last 2 merged pull requests in ${{ github.repository }}
  2. MCP Scripts GH CLI Testing: Use the mcpscripts-gh tool to query 2 pull requests from ${{ github.repository }} (use args: "pr list --repo ${{ github.repository }} --limit 2 --json number,title,author")
  3. Serena MCP Testing:
    • Use the Serena MCP server tool activate_project to initialize the workspace at ${{ github.workspace }} and verify it succeeds (do NOT use bash to run go commands - use Serena's MCP tools)
    • After initialization, use the find_symbol tool to search for symbols (find which tool to call) and verify that at least 3 symbols are found in the results
  4. Playwright Testing: Use the playwright tools to navigate to https://github.com and verify the page title contains "GitHub" (do NOT try to install playwright - use the provided MCP tools)
  5. Web Fetch Testing: Use the web-fetch tool to fetch https://github.com and verify the response contains "GitHub" (do NOT use bash or playwright for this test - use the web-fetch tool directly)
  6. File Writing Testing: Create a test file /tmp/gh-aw/agent/smoke-test-copilot-${{ github.run_id }}.txt with content "Smoke test passed for Copilot at $(date)" (create the directory if it doesn't exist)
  7. Bash Tool Testing: Execute bash commands to verify file creation was successful (use cat to read the file back)
  8. Discussion Interaction Testing:
    • Use the github-discussion-query mcp-script tool with params: limit=1, jq=".[0]" to get the latest discussion from ${{ github.repository }}
    • Extract the discussion number from the result (e.g., if the result is {"number": 123, "title": "...", ...}, extract 123)
    • Use the add_comment tool with discussion_number: <extracted_number> to add a fun, playful comment stating that the smoke test agent was here
  9. Build gh-aw: Run GOCACHE=/tmp/go-cache GOMODCACHE=/tmp/go-mod make build to verify the agent can successfully build the gh-aw project (both caches must be set to /tmp because the default cache locations are not writable). If the command fails, mark this test as ❌ and report the failure.
  10. Upload gh-aw binary as artifact: After a successful build, use bash to copy the ./gh-aw binary into the staging directory (mkdir -p $RUNNER_TEMP/gh-aw/safeoutputs/upload-artifacts && cp ./gh-aw $RUNNER_TEMP/gh-aw/safeoutputs/upload-artifacts/gh-aw), then call the upload_artifact safe-output tool with path: "gh-aw", retention_days: 1, and skip_archive: true. The upload_artifact tool is available and configured in this workflow run — use it directly, do NOT use missing_tool for it. Mark this test as ❌ if the build in step 9 failed.
  11. Discussion Creation Testing: Use the create_discussion safe-output tool to create a discussion in the announcements category titled "copilot was here" with the label "ai-generated"
  12. Workflow Dispatch Testing: Use the dispatch_workflow safe output tool to trigger the haiku-printer workflow with a haiku as the message input. Create an original, creative haiku about software testing or automation.
  13. PR Review Testing: Review the diff of the current pull request. Leave 1-2 inline create_pull_request_review_comment comments on specific lines, then call submit_pull_request_review with a brief body summarizing your review and event COMMENT. To test reply_to_pull_request_review_comment: use the pull_request_read tool (with method: "get_review_comments" and pullNumber: ${{ github.event.pull_request.number }}) to fetch the PR's existing review comments, then reply to the most recent one using reply_to_pull_request_review_comment with its actual numeric id as the comment_id. Note: create_pull_request_review_comment does not return a comment_id — you must fetch existing comment IDs from the GitHub API. If the PR has no existing review comments, skip the reply sub-test.

Output

  1. Create an issue with a summary of the smoke test run:

    • Use a temporary ID (e.g. aw_smoke1) for the issue so you can reference it later
    • Title: "Smoke Test: Copilot - ${{ github.run_id }}"
    • Body should include:
      • Test results (✅ or ❌ for each test)
      • Overall status: PASS or FAIL
      • Run URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
      • Timestamp
      • Pull request author and assignees
  2. Set Issue Type: Use the set_issue_type safe-output tool with issue_number: "aw_smoke1" (the temporary ID from step 1) and issue_type: "Bug" to set the type of the just-created smoke test issue.

  3. Only if this workflow was triggered by a pull_request event: Use the add_comment tool to add a very brief comment (max 5-10 lines) to the triggering pull request (omit the item_number parameter to auto-target the triggering PR) with:

    • PR titles only (no descriptions)
    • ✅ or ❌ for each test result
    • Overall status: PASS or FAIL
    • Mention the pull request author and any assignees
  4. Use the add_comment tool to add a fun and creative comment to the latest discussion (using the discussion_number you extracted in step 8) - be playful and entertaining in your comment

  5. Use the send_slack_message tool to send a brief summary message (e.g., "Smoke test ${{ github.run_id }}: All tests passed! ✅")

If all tests pass and this workflow was triggered by a pull_request event:

  • Use the add_labels safe-output tool to add the label smoke-copilot to the pull request (omit the item_number parameter to auto-target the triggering PR)
  • Use the remove_labels safe-output tool to remove the label smoke from the pull request (omit the item_number parameter to auto-target the triggering PR)

Important: If no action is needed after completing your analysis, you MUST call the noop safe-output tool with a brief explanation. Failing to call any safe-output tool is the most common cause of safe-output workflow failures.

{"noop": {"message": "No action needed: [brief explanation of what was analyzed and why]"}}