Add PR gate tests for azd ai agent extension#8754
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a PR-gate functional test suite for the azure.ai.agents extension using the existing recording/playback infrastructure, so ARM-dependent init flows can be validated in CI without Azure credentials.
Changes:
- Adds
record-tagged functional tests covering Tier 0 (offline) and Tier 1 (playback)azd ai agentscenarios. - Introduces a small
recordproxyhelper to route extension HTTP + ARM SDK traffic through the recorder proxy when running record/playback builds. - Extends the extension CI workflow to build record-tagged binaries and run the new Tier 0 + Tier 1 tests in PRs.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/test/functional/ai_agent_test.go | Adds record-tagged functional tests for azd ai agent Tier 0 + Tier 1 playback coverage. |
| cli/azd/test/azdcli/cli.go | Adjusts mock credential server token expiry timestamp formatting for compatibility with extension parsing. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport.go | Defines recordproxy.Transport as nil in non-record builds. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport_record.go | Implements proxy-aware HTTP transport + sets http.DefaultTransport in record builds. |
| cli/azd/extensions/azure.ai.agents/internal/pkg/azure/client_options.go | Injects the recording proxy transport into ARM SDK client options when present. |
| .github/workflows/lint-ext-azure-ai-agents.yml | Adds a PR job to build record-tagged azd/extension and run Tier 0 + Tier 1 playback tests. |
d6ae7c8 to
030a36d
Compare
2d83046 to
8a0b29a
Compare
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
…rtions - Equal-length sanitize all real subscription/RG/account/project/email/objectId in cassette (content_length unchanged, playback safe) - Sync test constants to match sanitized cassette values exactly - Add back pull-requests: write (required by reusable lint-go.yml) - Fix --model-deployment comment to document actual routing logic (flag existence routes to 'existing' branch, not value) - Strengthen MissingFlags assertion (check error message, not just exit code) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
e199073 to
04a9670
Compare
jongio
left a comment
There was a problem hiding this comment.
Solid work on the recording/playback test infrastructure. The tiered approach (Tier 0 offline, Tier 1 playback) is clean and the negative control test is a nice touch for proving cassette consumption. Cassette sanitization looks good (zeroed subscription IDs, fake resource names, sanitized auth headers).
A few observations:
Architecture (positive):
- Splitting the workflow into lint vs. test is the right call. The lint workflow (
lint-go.ymlreusable) needspull-requests: writefor annotations; the test workflow only needscontents: read. Clean separation of concerns. - CODEOWNERS entry for the test directory means the AI agents team owns these without requiring core team signoff. Good.
- Using
//go:build recordensures these tests never run accidentally in normalgo test ./...without the tag.
One thing to consider:
transport_record.goinit() replaceshttp.DefaultTransportglobally when therecordbuild tag is active. The comment on L41 acknowledges this covershttp.Client{}usage, andclient_options.goseparately handles Azure SDK clients. This is fine for the current test infrastructure, but if future code in the extension creates HTTP clients via other patterns (e.g., third-party SDKs that accept a custom transport parameter), those would silently bypass the proxy. Worth a brief comment in the package doc noting "all HTTP traffic is intercepted when this tag is active" so future contributors aren't surprised.
Re: open reviewer threads:
- trangevi's question about nesting: the tests already moved to
cli/azd/test/functional/ai_agents/with CODEOWNERS, which addresses the core concern. Further nesting would add complexity without clear benefit since Go packages are already the boundary. - trangevi's question about the lint workflow edit: the PR is splitting the pre-existing combined
ext-azure-ai-agents-ciworkflow into two focused workflows (lint + test). The lint job isn't new; it's the original workflow renamed. The test job is genuinely new. The commit message on HEAD explains this clearly.
…ODEOWNERS - Move test files to cli/azd/test/functional/ai_agents/ so the AI agents team owns them without requiring azd core team signoff on changes. - Add CODEOWNERS entry for the new test directory. - Split the combined lint+test workflow into two parallel workflows: - lint-ext-azure-ai-agents.yml (lint only, needs pull-requests: write) - test-ext-azure-ai-agents.yml (tests only, contents: read suffices) - Add helpers_test.go with shared test utilities for the new package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
04a9670 to
1acaafe
Compare
|
Thanks for the thorough review @jongio ! Added a package doc comment to |
|
Proposal: replace the manual install with the official The current "Install extension" step copies the binary into Suggested approach — use the developer extension ( - name: Install azd developer extension
run: |
azd extension source add -n azd -t url -l "https://aka.ms/azd/extensions/registry" # if not already present
azd extension install microsoft.azd.extensions
- name: Build, pack & publish extension to local registry (record-tagged)
working-directory: cli/azd/extensions/azure.ai.agents
env:
GOFLAGS: -tags=record # injected into build.sh's `go build`; no code change needed
run: |
azd x build # build.sh inherits GOFLAGS -> go build gets -tags=record; copies binary into ~/.azd/extensions/<id>/
azd x pack
azd x publish # registers the extension in the local registry from extension.yamlWhy this works for the Benefits
|
|
Test location: current structure is fine as step 1; the direction is to move tests into the extension folder I dug into why these functional tests live in Findings (for context):
Direction (tracked in #8795): expose a public, internal-free test-support module (its own No action needed in this PR for test location — just flagging the agreed direction so we migrate these into |
…ents - SampleList and SampleList_JSON were labeled Tier 0 (offline) but made live HTTPS calls to aka.ms/foundry-agents-samples. Move them to Tier 1 with recording sessions so they replay from cassette deterministically. - Remove hardcoded source line numbers from comments (they drift). - Tier 0 now has 4 truly offline tests; Tier 1 has 5 recorded tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2980481 to
b6f6721
Compare
Use the official azd extension developer workflow (azd x build/pack/publish) instead of hand-writing config.json via an inline Python heredoc. Benefits: - All extension metadata comes from extension.yaml — no hardcoded dict, no drift. - Exercises azd's real install/registration path. - Removes the brittle YAML-embedded Python script. The record build tag is injected via GOFLAGS=-tags=record, which build.sh inherits through os.Environ() (verified locally and by the reviewer). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
b6f6721 to
e4f8b19
Compare
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
|
@vhvb1989 Simplifying install is done — switched to |
jongio
left a comment
There was a problem hiding this comment.
Clean refactoring that addresses all prior feedback. Tests properly isolated in their own subdirectory and package, workflows split with correct permissions, CODEOWNERS added. LGTM.
danieljurek
left a comment
There was a problem hiding this comment.
I think this is a good first cut at narrowly testing this single extension. If you want to test other extensions in this way, I recommend a few things:
- Use one workflow to test all extensions (figure out which extension to test based on files in the diff)
- Use ci-build.ps1 in each extension and add support for the tags/params you want to use in that build
- Use ci-test.ps1 if that makes sense in this case instead of calling go test directly... because these tests are running from outside the extension directory, perhaps that may not be applicable here.
Four recurring themes promoted from PR reviews merged since 2026-06-22: 1. go.instructions.md: Test goroutine safety — t.Fatal/require.NoError must not be called from httptest handler goroutines or other non-test goroutines; HTTP headers must be set before WriteHeader. (Source: #8790) 2. extensions.instructions.md: Duplicate helper detection — export shared helpers from the owning package instead of copying function bodies across extension packages. (Source: #8794, #8809) 3. extensions.instructions.md: User-provided path validation — paths from azure.yaml fields (instructions, entryPoint, etc.) must be validated to stay within the expected root directory. (Source: #8779, #8599) 4. extensions.instructions.md: Cross-extension test isolation — tests loading sibling-extension files must skip when absent; extension functional tests should have dedicated CODEOWNERS entries. (Source: #8809, #8754) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes #8760
Summary
Adds automated PR gate tests for the
azd ai agentextension using azd's existing recording/playback proxy infrastructure. These tests run in CI on every PR touching the extension, catching regressions without requiring live Azure credentials.Test Architecture
4-Process Recording Chain
The recording proxy intercepts all outbound HTTP from the extension, storing request/response pairs in YAML cassette files. During playback, the proxy replays stored responses — no network access needed.
Test Tiers
AZURE_RECORD_MODE=playbackinit --no-promptwith ARM calls replayed from cassette — model catalog queries, project resolution, scaffold generationTests (9 total, all passing)
Tier 0 — Offline (6 tests):
Test_AIAgent_Version— extension version commandTest_AIAgent_Help— help outputTest_AIAgent_Init_NoPrompt_MissingFlags— missing required flags → proper errorTest_AIAgent_SampleList— sample template listingTest_AIAgent_SampleList_JSON— JSON output with proper structure validation (json.Unmarshal + key check)Test_AIAgent_Doctor_Help— doctor subcommand helpTier 1 — Recording Playback (3 tests):
Test_AIAgent_Init_NoPrompt_Defer— init with--deploy-mode defer(no ARM calls, scaffold only)Test_AIAgent_Init_NoPrompt_WithProject— full init with--project-id, resolves Foundry project via ARM, auto-selects model from catalog, generates complete scaffoldTest_AIAgent_Init_NegativeControl_BadCassette— negative control: empty cassette → init fails with "requested interaction not found", proving the cassette is genuinely consumedKey Assertions (WithProject)
azure.yaml,agent.yaml, agent source directory.envfile containsAZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4.1"(resolved from cassette's model catalog, not a placeholder)agent.yamlhas resolved value (not${...}env placeholder)Files Changed
New files
cli/azd/test/functional/ai_agents/ai_agent_test.gocli/azd/test/functional/ai_agents/ai_agent_recording_test.gocli/azd/test/functional/ai_agents/testdata/manifests/foundry-toolbox.yamlcli/azd/test/functional/ai_agents/testdata/recordings/Test_AIAgent_Init_NoPrompt_WithProject.yamlcli/azd/test/functional/ai_agents/testdata/recordings/Test_AIAgent_Init_NoPrompt_Defer.yamlcli/azd/test/functional/ai_agents/testdata/recordings/Test_AIAgent_Init_NegativeControl_BadCassette.yamlcli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport.gocli/azd/extensions/azure.ai.agents/internal/pkg/recordproxy/transport_record.gohttp.DefaultTransportto route through proxy.github/workflows/lint-ext-azure-ai-agents.ymlModified files
cli/azd/extensions/azure.ai.agents/internal/pkg/azure/client_options.goproxyTransporterstruct —*http.Clientdirectly satisfiespolicy.Transporterinterfacecli/azd/test/azdcli/cli.go+00:00fromtime.RFC3339)cli/azd/extensions/azure.ai.agents/cspell.yamlrecordproxyto dictionaryCI Workflow Design
Key CI considerations:
azdis missing (Tier 0). Withazdpresent but not logged in → hard fail. The symlink is only created for Tier 1 where fake auth via recording session works.-mflag points to committedtestdata/manifests/foundry-toolbox.yamlinstead of GitHub URL, avoiding device-code auth in clean CI.~/.azd/config.jsonwith extension path, matching azd's discovery mechanism.Cassette Size
The
WithProjectcassette is 2.4 MB (47 interactions). For context, the repo's existing 29 cassettes have median 0.97 MB and max 4.80 MB — this is within normal range. The cassette was trimmed from 26 MB by keeping only target models (gpt-4o, gpt-4.1, gpt-4o-mini) in model catalog responses.How to Re-record
The cassette file is updated in-place. Trim if needed (model catalog responses are large).
New CI Workflow
This PR adds a new workflow that takes effect immediately after merge to main:
Trigger: pull_request when changes are made to any of:
In short: any PR that touches the AI agent extension code, test files, recordings, or the workflow itself will automatically run lint, extension build, and the full test suite.
Test Cases at a Glance