perf: reduce smoke-copilot token usage with pre-steps and tool trimming#1613
perf: reduce smoke-copilot token usage with pre-steps and tool trimming#1613
Conversation
- Remove unused tools: agentic-workflows, cache-memory, edit, playwright, web-fetch (56 tools → ~10, saving ~30K tokens/request from schema overhead) - Remove unused network groups: node, playwright - Restrict github toolsets to [repos, pull_requests] - Add pre-agent step to deterministically pre-compute: - Last 2 merged PRs via gh CLI - GitHub.com connectivity check via curl - File write/read test - Simplify agent prompt to verify pre-computed results and format output - Replace Playwright browser test with curl connectivity check in pre-step - Fix safe-outputs YAML indentation (was over-indented) - Remove sandbox.mcp.container (not allowed in strict mode) Expected token reduction: ~60-70% per run by eliminating tool probing turn, reducing tool schema overhead, and moving deterministic work out of the LLM context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
Smoke Test Results
Overall: PASS
|
There was a problem hiding this comment.
Pull request overview
This PR updates the Smoke Copilot agentic workflow to significantly reduce token usage by shrinking the tool surface, moving deterministic checks into pre-agent steps, and simplifying the agent prompt.
Changes:
- Reduced configured MCP/tool surface (removed unused tool groups; restrict GitHub MCP toolsets).
- Added a pre-agent
steps:block to pre-compute smoke-test data (merged PR list, github.com connectivity, file IO check) and reference results in the prompt. - Updated the compiled workflow lockfile to gh-aw v0.65.3, reflecting the new tool/network configuration and safe-outputs wiring.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/smoke-copilot.md | Trims tools/network, adds pre-agent steps, and rewrites the prompt to use pre-computed results. |
| .github/workflows/smoke-copilot.lock.yml | Regenerates the compiled workflow with the new configuration and updated gh-aw version. |
Comments suppressed due to low confidence (1)
.github/workflows/smoke-copilot.lock.yml:345
- The
smoke-datastep runs in theagentjob and sets step outputs, but the prompt artifact is generated earlier inactivation. As a result, the step outputs can’t influence the agent prompt (and other jobs can’t consume them unless written to a file/artifact). Prefer persisting the computed results to a known file under /tmp/gh-aw and referencing that file in the prompt, or relocate smoke-data into the activation job before prompt generation.
- name: Configure gh CLI for GitHub Enterprise
run: bash ${RUNNER_TEMP}/gh-aw/actions/configure_gh_for_ghe.sh
env:
GH_TOKEN: ${{ github.token }}
- env:
GH_TOKEN: ${{ github.token }}
id: smoke-data
name: Pre-compute smoke test data
run: "echo \"::group::Fetching last 2 merged PRs\"\nPR_DATA=$(gh pr list --repo \"$GITHUB_REPOSITORY\" --state merged --limit 2 \\\n --json number,title,author,mergedAt \\\n --jq '.[] | \"PR #\\(.number): \\(.title) (by @\\(.author.login), merged \\(.mergedAt))\"')\necho \"$PR_DATA\"\necho \"::endgroup::\"\n\necho \"::group::GitHub.com connectivity check\"\nHTTP_CODE=$(curl -s -o /dev/null -w \"%{http_code}\" --max-time 10 https://github.com)\necho \"github.com returned HTTP $HTTP_CODE\"\necho \"::endgroup::\"\n\necho \"::group::File write/read test\"\nTEST_DIR=\"/tmp/gh-aw/agent\"\nTEST_FILE=\"$TEST_DIR/smoke-test-copilot-${GITHUB_RUN_ID}.txt\"\nmkdir -p \"$TEST_DIR\"\necho \"Smoke test passed for Copilot at $(date)\" > \"$TEST_FILE\"\nFILE_CONTENT=$(cat \"$TEST_FILE\")\necho \"Wrote and read back: $FILE_CONTENT\"\necho \"::endgroup::\"\n\n# Export results for agent context\n{\n echo \"SMOKE_PR_DATA<<SMOKE_EOF\"\n echo \"$PR_DATA\"\n echo \"SMOKE_EOF\"\n echo \"SMOKE_HTTP_CODE=$HTTP_CODE\"\n echo \"SMOKE_FILE_CONTENT=$FILE_CONTENT\"\n echo \"SMOKE_FILE_PATH=$TEST_FILE\"\n} >> \"$GITHUB_OUTPUT\"\n"
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ### 1. GitHub MCP Testing | ||
| The last 2 merged pull requests have been fetched. Verify MCP connectivity by calling `github-list_pull_requests` for ${{ github.repository }} (limit 1, state merged) and confirm data is returned. | ||
|
|
||
| ### 2. GitHub.com Connectivity | ||
| Pre-step result: HTTP ${{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }} from github.com. | ||
| ✅ if HTTP 200 or 301, ❌ otherwise. | ||
|
|
||
| ### 3. File Write/Read Test | ||
| Pre-step wrote and read back: "${{ steps.smoke-data.outputs.SMOKE_FILE_CONTENT }}" | ||
| File path: ${{ steps.smoke-data.outputs.SMOKE_FILE_PATH }} | ||
| Verify by running `cat` on the file path using bash to confirm it exists. | ||
|
|
||
| ### 4. Bash Tool Testing | ||
| Run a simple bash command (e.g., `echo "bash works"`) to verify the bash tool is functional. | ||
|
|
||
| ## Pre-Fetched PR Data | ||
|
|
||
| 1. **GitHub MCP Testing**: Review the last 2 merged pull requests in ${{ github.repository }} | ||
| 2. **Playwright Testing**: Use playwright to navigate to https://github.com and verify the page title contains "GitHub" | ||
| 3. **File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-copilot-${{ github.run_id }}.txt` with content "Smoke test passed for Copilot at $(date)" (create the directory if it doesn't exist) | ||
| 4. **Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back) | ||
| ``` | ||
| ${{ steps.smoke-data.outputs.SMOKE_PR_DATA }} | ||
| ``` |
There was a problem hiding this comment.
The prompt references ${{ steps.smoke-data.outputs.* }} (HTTP code, file content/path, PR data). In gh-aw compiled workflows the prompt is rendered during the activation job, before the agent job runs steps:; these steps.* outputs will be empty and the agent will see missing data. Consider writing the pre-computed results to a deterministic file path (e.g., /tmp/gh-aw/smoke-data.env) and instructing the agent to cat it, or move the pre-compute step into the same job that renders the prompt (activation) so the values can be injected.
See below for a potential fix:
A pre-step has stored the HTTP status code from github.com in `/tmp/gh-aw/smoke-data.env` under the key `SMOKE_HTTP_CODE`.
Use the bash tool to read this value (for example, by running `cat /tmp/gh-aw/smoke-data.env`) and interpret it as:
✅ if HTTP 200 or 301, ❌ otherwise.
### 3. File Write/Read Test
The same pre-step recorded `SMOKE_FILE_CONTENT` and `SMOKE_FILE_PATH` in `/tmp/gh-aw/smoke-data.env`.
Use bash to read these values from the env file, then run `cat` on the reported file path to confirm the file exists and contains the expected content.
### 4. Bash Tool Testing
Run a simple bash command (e.g., `echo "bash works"`) to verify the bash tool is functional.
## Pre-Fetched PR Data
A pre-step has written pre-fetched pull request data to `/tmp/gh-aw/smoke-pr-data.json`.
Use bash to inspect this file (for example, by running `cat /tmp/gh-aw/smoke-pr-data.json`) and use that data when summarizing PR titles and statuses.
| GH_AW_EXPR_2805DAC9: ${{ steps.smoke-data.outputs.SMOKE_FILE_PATH }} | ||
| GH_AW_EXPR_7EA93000: ${{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }} | ||
| GH_AW_EXPR_ABDF8D58: ${{ steps.smoke-data.outputs.SMOKE_PR_DATA }} | ||
| GH_AW_EXPR_EC16C26C: ${{ steps.smoke-data.outputs.SMOKE_FILE_CONTENT }} |
There was a problem hiding this comment.
In the activation job, Create prompt with built-in context exports GH_AW_EXPR_* values from ${{ steps.smoke-data.outputs.* }}, but there is no smoke-data step in the activation job. This will render the prompt with empty placeholders. To fix, run the smoke-data step in activation before prompt creation (or stop injecting step outputs into the prompt and have the agent read results from a file created in the agent job).
This issue also appears on line 336 of the same file.
| GH_AW_EXPR_2805DAC9: ${{ steps.smoke-data.outputs.SMOKE_FILE_PATH }} | |
| GH_AW_EXPR_7EA93000: ${{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }} | |
| GH_AW_EXPR_ABDF8D58: ${{ steps.smoke-data.outputs.SMOKE_PR_DATA }} | |
| GH_AW_EXPR_EC16C26C: ${{ steps.smoke-data.outputs.SMOKE_FILE_CONTENT }} | |
| GH_AW_EXPR_2805DAC9: "" | |
| GH_AW_EXPR_7EA93000: "" | |
| GH_AW_EXPR_ABDF8D58: "" | |
| GH_AW_EXPR_EC16C26C: "" |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
The gh-aw v0.65+ compiler no longer sets GH_AW_SAFE_OUTPUTS as a
job-level env var (it uses step outputs instead). The post-step
validation was falling back to /opt/gh-aw/safeoutputs/outputs.jsonl
which doesn't exist. Use ${RUNNER_TEMP}/gh-aw/safeoutputs/outputs.jsonl
as the fallback instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Smoke Test Results — PASS✅ GitHub MCP: PR #1517 "fix: prevent Squid config injection via --allow-domains and --allow-urls", PR #1446 "fix: add picomatch overrides for GHSA-c2c7-rcm5-vvqj" Overall: PASS
|
🏗️ Build Test Suite Results
Overall: 0/8 ecosystems passed — ❌ FAIL Error: All repository clones failed because
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
The post-processor's checkout injection logic was fooled by the conditional 'Checkout repository for patch context' step in the detection job. Since it has an 'if:' guard, it may be skipped, leaving npm ci without a package-lock.json. Fix the regex to skip conditional checkouts (those with an 'if:' on the next line) so an unconditional checkout is properly injected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Smoke Test Results
Overall: PASS
|
🔥 Smoke Test Results
Overall: PASS PR by @lpcox · Reviewer:
|
Smoke Test: GitHub Actions Services Connectivity ✅
All checks passed. (
|
|
Smoke Test Results (Codex)
|
Summary
Reduces smoke-copilot token usage by an estimated 60-70% per run through four optimizations identified in #1604.
Changes
1. Tool Surface Reduction (est. ~30K tokens saved)
Removed 6 unused tool groups, keeping only what the workflow actually uses:
agentic-workflows,cache-memory,edit,playwright,web-fetchbash,github(restricted to[repos, pull_requests]toolsets)2. Pre-Agent Steps (moves work out of LLM context)
Added a
steps:block that runs before the agent, outside the firewall sandbox:gh pr list(deterministic, no LLM needed)curl(replaces Playwright browser test)3. Simplified Agent Prompt
Agent now just needs to:
4. Network & Config Cleanup
node,playwrightsandbox.mcp.container(not allowed in strict mode)safe-outputsYAML indentation (was incorrectly nested undersandbox)Token Impact Analysis
Ref: #1604