feat: wire BYOK providers by slbug · Pull Request #240 · microsoft/waza

slbug · 2026-05-16T11:07:26Z

Summary

Adds a clean custom-provider lane for the Copilot SDK executor:

passes COPILOT_* provider env vars into session create/resume
skips default Copilot auth checks when a custom provider is configured
labels custom-provider request counts as Provider Requests
records only sanitized provider metadata in result JSON
updates docs for the new env vars and usage semantics

Also disables GPG signing in temp git repos used by token tests.

Validation

go test ./...
cd site && npm ci --include=optional
cd site && npm run build

slbug · 2026-05-16T11:09:41Z

@microsoft-github-policy-service agree

codecov-commenter · 2026-05-20T19:42:52Z

Codecov Report

❌ Patch coverage is 87.50000% with 10 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@2f0f6d9). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
cmd/waza/cmd_run.go	36.36%	7 Missing ⚠️
internal/execution/copilot.go	94.54%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #240   +/-   ##
=======================================
  Coverage        ?   75.81%           
=======================================
  Files           ?      154           
  Lines           ?    18014           
  Branches        ?        0           
=======================================
  Hits            ?    13657           
  Misses          ?     3399           
  Partials        ?      958

Flag	Coverage Δ
go-implementation	`75.81% <87.50%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Adds a “custom/BYOK provider” lane to the copilot-sdk executor so Waza can route sessions through non-default Copilot SDK providers, label usage appropriately, and persist only sanitized provider metadata in results.

Changes:

Thread COPILOT_* provider environment configuration into Copilot SDK session create/resume and skip default Copilot auth checks when a custom provider is configured.
Extend usage stats to record provider / provider_host, preserve them on aggregation, and update CLI usage labeling (e.g., “Provider Requests”).
Update docs to describe the new provider env vars and tweak token tests’ temp git repo setup to disable GPG signing.

Show a summary per file

File	Description
site/src/content/docs/reference/cli.mdx	Documents custom provider env vars + clarifies auth requirement is for the default provider.
site/src/content/docs/quick-start.mdx	Updates quick start auth/prereqs; adds note about custom provider configuration.
README.md	Adds custom provider section and updates auth wording in CI guidance.
internal/models/outcome.go	Adds provider metadata fields/constants and preserves them during usage aggregation.
internal/models/outcome_test.go	Adds tests covering provider metadata preservation and “mixed” aggregation behavior.
internal/execution/copilot.go	Parses provider config from env, skips auth check in BYOK mode, passes provider config to sessions, and annotates usage with sanitized provider host.
internal/execution/copilot_test.go	Adds tests for BYOK auth-skip, provider pass-through on create/resume, and host sanitization.
docs/SKILLS_CI_INTEGRATION.md	Clarifies default route uses `GITHUB_TOKEN`, and custom providers use `COPILOT_*` vars instead.
docs/GETTING-STARTED.md	Updates mock executor guidance to mention custom provider configuration.
cmd/waza/tokens/internal/git/git_test.go	Disables GPG signing in temp repos used by token tests.
cmd/waza/tokens/compare_test.go	Disables GPG signing in temp repos used by token tests.
cmd/waza/cmd_run.go	Prints provider metadata and relabels request counters based on provider route.

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 1

Copilot

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 1

spboyer

Reviewed the BYOK provider wiring and docs; the implementation looks sound, but two README auth/provider examples need correction before merge.

Issues to address:

README.md:381 - waza models still lists models through the default Copilot API, so BYOK auth bypass wording is too broad
README.md:1173 - Azure OpenAI example uses OpenAI provider/path and an unsupported wire API value

spboyer

The current PR head fixes the README issues from my prior review. One site CLI reference page still has the same waza models custom-provider auth wording problem.

Issues to address:

site/src/content/docs/reference/cli.mdx:909 - waza models still requires Copilot auth and does not use the custom provider session config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer

The remaining CLI reference docs issue is fixed in 66fcf42. LGTM.

Copilot

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer

Addressed the remaining review feedback and resolved the threads. LGTM.

) (#306) * fix: skip --model CLI startup arg when BYOK provider is configured (#305) PR #263 added --model <defaultModelID> to copilot.ClientOptions.CLIArgs so the embedded Copilot CLI honors the eval-configured model. However, the CLI validates that startup --model against the GitHub Copilot catalog BEFORE the per-session ProviderConfig from #240 is applied. For BYOK / custom provider model IDs that are not in the Copilot catalog (e.g. minimax-m2.7), startup fails with 'Model "..." is not available'. Suppress the startup --model arg whenever providerFromEnv() reports an enabled custom provider. SessionConfig.Model + SessionConfig.Provider are still passed per session via Execute, so the SDK selects the right model against the custom provider without pre-validating against the GitHub catalog. Normal GitHub Copilot models retain the #263 behavior. Adds a regression test covering the BYOK + non-empty defaultModelID case, and hardens the existing CLIArgs tests against environments that already have COPILOT_BASE_URL set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(test): properly unset COPILOT_* env vars in clearCustomProviderEnv Replace t.Setenv(name, "") with os.Unsetenv + t.Cleanup so the helper actually removes variables from the environment rather than setting them to empty strings. This matches the comment ("unsets") and avoids any potential confusion for future readers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(test): ignore os.Setenv/Unsetenv return values in clearCustomProviderEnv Satisfies errcheck in golangci-lint v2.10.1; these calls don't realistically fail in tests and t.Cleanup ensures restoration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Shayne Boyer <spboyer@users.noreply.github.com>

slbug requested a review from spboyer as a code owner May 16, 2026 11:07

github-actions Bot enabled auto-merge (squash) May 16, 2026 11:08

spboyer requested a review from Copilot May 20, 2026 19:38

Copilot started reviewing on behalf of spboyer May 20, 2026 19:40 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread README.md Outdated

This was referenced May 20, 2026

[E1] Decouple ExecutionResponse from Copilot SDK + Multi-Agent Engine Support #10

Closed

Bring Your Own Model — Ollama, OpenAI, Anthropic, OpenCode engines #11

Open

spboyer requested a review from Copilot May 21, 2026 20:37

Copilot started reviewing on behalf of spboyer May 21, 2026 20:37 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread internal/execution/copilot.go Outdated

spboyer requested changes May 22, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md Outdated

feat: wire BYOK providers

7d6c940

auto-merge was automatically disabled May 22, 2026 21:03
Head branch was pushed to by a user without write access

slbug force-pushed the custom_provider branch from 76843e9 to 7d6c940 Compare May 22, 2026 21:03

slbug requested a review from spboyer May 22, 2026 21:11

spboyer requested changes May 23, 2026

View reviewed changes

Comment thread site/src/content/docs/reference/cli.mdx Outdated

docs: clarify models auth with custom providers

66fcf42

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 23, 2026 14:03

Copilot started reviewing on behalf of spboyer May 23, 2026 14:03 View session

spboyer approved these changes May 23, 2026

View reviewed changes

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread README.md Outdated

fix: address BYOK provider review feedback

5c88655

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer approved these changes May 23, 2026

View reviewed changes

spboyer enabled auto-merge (squash) May 23, 2026 14:09

spboyer merged commit 769fd04 into microsoft:main May 23, 2026
7 checks passed

eric-x-liu mentioned this pull request May 30, 2026

BYOK provider models fail after startup --model is passed to Copilot CLI #305

Closed

spboyer mentioned this pull request Jun 1, 2026

fix: skip --model CLI startup arg when BYOK provider is configured (#305) #306

Merged

Uh oh!

Conversation

slbug commented May 16, 2026

Summary

Validation

Uh oh!

slbug commented May 16, 2026

Uh oh!

codecov-commenter commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented May 20, 2026 •

edited

Loading