feat: add smoke-services workflow for --allow-host-service-ports e2e testing by lpcox · Pull Request #1534 · github/gh-aw-firewall

lpcox · 2026-03-31T23:44:23Z

Summary

Add a new smoke test workflow that validates the full --allow-host-service-ports path end-to-end:

services: port mappings → --allow-host-service-ports → iptables rules → agent reaches Redis/PostgreSQL

What's included

`.github/workflows/smoke-services.md`

New agentic workflow source:

Declares Redis (6379) and PostgreSQL (5432) as the services to test
Agent installs redis-tools and postgresql-client, then verifies connectivity
Validates PING/PONG for Redis, pg_isready + SELECT 1 for PostgreSQL

`.github/workflows/smoke-services.lock.yml`

Compiled + postprocessed lock file with:

GitHub Actions services: block (redis:7 on 6379, postgres:15 on 5432)
--allow-host-service-ports flag with ${{ job.services.*.ports[] }} expressions
Local build steps (via postprocess script)

`scripts/ci/postprocess-smoke-workflows.ts`

Registered the new lock file in the postprocess pipeline.

Testing strategy

This is a workflow-level test — it will be validated when manually triggered via workflow_dispatch. The workflow itself IS the test: if the agent can reach Redis and PostgreSQL through the firewall, the feature works.

Closes #1525

…testing Add a new smoke test workflow that validates the full path: services: port mappings → --allow-host-service-ports → iptables rules → agent reaches Redis and PostgreSQL on localhost The workflow: - Declares Redis (6379) and PostgreSQL (5432) as GitHub Actions services - Passes compiler-generated port expressions to --allow-host-service-ports - Agent installs redis-tools/postgresql-client and verifies connectivity - Validates PING/PONG for Redis, pg_isready + SELECT 1 for PostgreSQL Also registers the new lock file in postprocess-smoke-workflows.ts. Closes #1525 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-31T23:45:18Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	82.67%	82.77%	📈 +0.10%
Statements	82.34%	82.43%	📈 +0.09%
Functions	81.22%	81.22%	➡️ +0.00%
Branches	75.94%	76.00%	📈 +0.06%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	85.8% → 86.2% (+0.41%)	85.3% → 85.7% (+0.40%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Adds a new GitHub Actions smoke workflow intended to exercise the end-to-end --allow-host-service-ports path by bringing up Redis/PostgreSQL services: and verifying the AWF sandbox can reach them.

Changes:

Added new smoke workflow source (smoke-services.md) and compiled lock workflow (smoke-services.lock.yml) to validate service connectivity.
Registered the new lock workflow in the smoke-workflow postprocess pipeline.
Updated the actions lockfile to include the github/gh-aw-actions/setup@v0.64.5 entry used by the new lock workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
scripts/ci/postprocess-smoke-workflows.ts	Adds `smoke-services.lock.yml` to the postprocess list so it gets the same normalization as other smoke workflows.
.github/workflows/smoke-services.md	New agentic smoke workflow definition and instructions for Redis/PostgreSQL connectivity checks.
.github/workflows/smoke-services.lock.yml	Compiled workflow with `services:` containers and AWF invocation including `--allow-host-service-ports`.
.github/aw/actions-lock.json	Adds the pinned `github/gh-aw-actions/setup@v0.64.5` action entry referenced by the new lock workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-31T23:48:25Z

.github/workflows/smoke-services.md

+1. **Redis Connectivity**: Run `redis-cli -h localhost -p 6379 ping` and verify the response is `PONG`
+2. **PostgreSQL Connectivity**: Run `pg_isready -h localhost -p 5432` and verify it reports the server is accepting connections
+3. **PostgreSQL Query**: Run `PGPASSWORD=testpass psql -h localhost -p 5432 -U postgres -d smoketest -c "SELECT 1 AS smoke_test;"` and verify it returns a row


The connectivity commands use -h localhost, but inside the AWF sandbox localhost resolves to the sandbox container itself, not the GitHub Actions runner where the services: ports are published. This will likely cause the smoke test to fail even when --allow-host-service-ports is working. Use host.docker.internal (host gateway) for Redis/PostgreSQL connections so the test exercises the intended host-service-port path.

Suggested change

1. **Redis Connectivity**: Run `redis-cli -h localhost -p 6379 ping` and verify the response is `PONG`

2. **PostgreSQL Connectivity**: Run `pg_isready -h localhost -p 5432` and verify it reports the server is accepting connections

3. **PostgreSQL Query**: Run `PGPASSWORD=testpass psql -h localhost -p 5432 -U postgres -d smoketest -c "SELECT 1 AS smoke_test;"` and verify it returns a row

1. **Redis Connectivity**: Run `redis-cli -h host.docker.internal -p 6379 ping` and verify the response is `PONG`

2. **PostgreSQL Connectivity**: Run `pg_isready -h host.docker.internal -p 5432` and verify it reports the server is accepting connections

3. **PostgreSQL Query**: Run `PGPASSWORD=testpass psql -h host.docker.internal -p 5432 -U postgres -d smoketest -c "SELECT 1 AS smoke_test;"` and verify it returns a row

github-actions · 2026-03-31T23:49:16Z

Smoke Test Results — Claude Engine

Test	Result
GitHub MCP: #1528 fix: api-proxy auth chain — trim keys, align placeholder format, add diagnostics	✅
GitHub MCP: #1508 fix: copy get-claude-key.sh to chroot-accessible path	✅
Playwright: github.com title contains "GitHub"	✅
File write + bash verify	✅

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude for issue #1534

github-actions · 2026-03-31T23:49:49Z

Smoke Test Results — Copilot Engine ✅ PASS

Test	Result
GitHub MCP — last 2 merged PRs	✅ #1528 `fix: api-proxy auth chain…`, #1508 `fix: copy get-claude-key.sh…`
Playwright — github.com title	✅ Contains "GitHub"
File write `/tmp/gh-aw/agent/smoke-test-copilot-23824680063.txt`	✅
Bash verify (`cat`)	✅

Overall: PASS — @lpcox (no assignees)

📰 BREAKING: Report filed by Smoke Copilot for issue #1534

Two CI fixes: 1. Agent connects to host.docker.internal instead of localhost (localhost inside the AWF container is the container itself, not the host where GitHub Actions services run) 2. Add GH_AW_SAFE_OUTPUTS env var to the validate step so it reads the outputs file from the correct v0.64.5 path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-01T00:18:44Z

Smoke Test Results — PASS

Test	Result
GitHub MCP: #1528 fix: api-proxy auth chain	✅
GitHub MCP: #1508 fix: copy get-claude-key.sh to chroot-accessible path	✅
Playwright: github.com title contains "GitHub"	✅
File write + bash verify	✅

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude for issue #1534

github-actions · 2026-04-01T00:18:51Z

Smoke Test Results — Run 23825478676

✅ GitHub MCP — Last 2 merged PRs: #1528 "fix: api-proxy auth chain…" · #1508 "fix: copy get-claude-key.sh…" (author: @lpcox, no assignees)
✅ Playwright — github.com title contains "GitHub"
✅ File write — /tmp/gh-aw/agent/smoke-test-copilot-23825478676.txt created and verified
✅ Bash — file read back successfully

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot for issue #1534

Rewrite the test instructions to be less prescriptive about output format. The previous prompt explicitly described PASS/FAIL format and tool actions (add comment, add label), which the threat detection model misinterpreted as a pre-fabricated injection payload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-01T00:35:07Z

Smoke test results (run 23825943785)

✅ GitHub MCP — #1528 fix: api-proxy auth chain; #1508 fix: copy get-claude-key.sh to chroot-accessible path
✅ Playwright — github.com title contains "GitHub"
✅ File write — smoke-test-claude-23825943785.txt created
✅ Bash verify — file contents confirmed

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude for issue #1534

github-actions · 2026-04-01T00:35:10Z

Smoke Test Results

Test	Status
GitHub MCP (last 2 merged PRs)	✅
Playwright (github.com title check)	✅
File write	✅
Bash verification	✅

Last 2 merged PRs: #1528 "fix: api-proxy auth chain — trim keys, align placeholder format, add diagnostics" · #1508 "fix: copy get-claude-key.sh to chroot-accessible path" (both by @lpcox)

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot for issue #1534

github-actions · 2026-04-01T00:36:48Z

Smoke Test: GitHub Actions Services Connectivity ✅

Check	Result
Redis PING (`host.docker.internal:6379`)	✅ `PONG`
PostgreSQL ready (`host.docker.internal:5432`)	✅ accepting connections
`SELECT 1` on `smoketest` DB as `postgres`	✅ returned `1`

All checks passed.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-04-01T00:37:10Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #1534 · ◷

github-actions · 2026-04-01T01:11:12Z

PRs reviewed: "fix: api-proxy auth chain — trim keys, align placeholder format, add diagnostics" | "fix: copy get-claude-key.sh to chroot-accessible path"
GitHub MCP review ✅
safeinputs-gh PR query ❌ (tool unavailable; used GitHub MCP fallback)
Playwright title check ❌ (EACCES from Playwright MCP)
Tavily search ❌ (Tavily MCP unavailable)
File write/read + bash cat ✅
Discussion oracle comment ✅
Build (npm ci && npm run build) ✅
Overall: FAIL

🔮 The oracle has spoken through Smoke Codex

lpcox requested a review from Mossaka as a code owner March 31, 2026 23:44

Copilot AI review requested due to automatic review settings March 31, 2026 23:44

Copilot started reviewing on behalf of lpcox March 31, 2026 23:45 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

github-actions bot added the smoke-claude label Mar 31, 2026

github-actions bot added the smoke-copilot label Mar 31, 2026

This comment has been minimized.

Sign in to view

This was referenced Mar 31, 2026

[aw] Smoke Services failed #1535

Closed

[agentics] No-Op Runs #1523

Closed

This comment has been minimized.

Sign in to view

github-actions bot added the build-test label Apr 1, 2026

This comment has been minimized.

Sign in to view

github-actions bot added the smoke-services label Apr 1, 2026

This comment has been minimized.

Sign in to view

lpcox merged commit e31b23e into main Apr 1, 2026
60 of 61 checks passed

lpcox deleted the copilot/smoke-services-test branch April 1, 2026 01:08

This was referenced Apr 1, 2026

feat: add token usage tracking to api-proxy sidecar #1539

Merged

fix: disable IPv6 in agent container to prevent squid proxy bypass #1544

Merged

Conversation

lpcox commented Mar 31, 2026

Summary

What's included

.github/workflows/smoke-services.md

.github/workflows/smoke-services.lock.yml

scripts/ci/postprocess-smoke-workflows.ts

Testing strategy

Uh oh!

github-actions bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 31, 2026

Uh oh!

github-actions bot commented Mar 31, 2026

Smoke Test Results — Copilot Engine ✅ PASS

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Apr 1, 2026

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Apr 1, 2026

Uh oh!

github-actions bot commented Apr 1, 2026

Smoke Test Results

Uh oh!

github-actions bot commented Apr 1, 2026

Smoke Test: GitHub Actions Services Connectivity ✅

Uh oh!

github-actions bot commented Apr 1, 2026

🏗️ Build Test Suite Results

Uh oh!

This comment has been minimized.

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`.github/workflows/smoke-services.md`

`.github/workflows/smoke-services.lock.yml`

`scripts/ci/postprocess-smoke-workflows.ts`

github-actions bot commented Mar 31, 2026 •

edited

Loading