Summary

copilot-sdk evals using a custom/BYOK provider fail on current main when --model is a provider-only model ID (for example minimax-m2.7).

This appears to be an interaction between:

• feat: wire BYOK providers #240, which correctly wires COPILOT_BASE_URL / COPILOT_PROVIDER_BASE_URL into SessionConfig.Provider, and
• fix: pass --model via CLIArgs to override user settings and experiment flights #263, which also passes the eval model to the embedded Copilot CLI process as startup CLIArgs: ["--model", defaultModelID].

The startup --model is validated by the Copilot CLI before the per-session ProviderConfig is applied. For custom-provider model IDs that are not in the GitHub Copilot catalog, startup fails with Model "..." is not available, even though the same model works when supplied as SessionConfig.Model with SessionConfig.Provider.

Repro

Using a LiteLLM/OpenAI-compatible endpoint that exposes minimax-m2.7:

COPILOT_BASE_URL=https://api.example.com/v1 \
COPILOT_PROVIDER=openai \
COPILOT_API_KEY=REDACTED \
waza run /path/to/eval.yaml --model minimax-m2.7 -v

Observed on current origin/main (23e9dbae) and v0.33-era builds after #263:

failed to create session: failed to create session: JSON-RPC Error -32603:
Request session.create failed with message: Model "minimax-m2.7" is not available.

Investigation

I isolated the behavior across Waza commits:

v0.31.0      works
f65915c1     works  # Copilot SDK v0.3.0 migration
508462dd     fails before #240 because ProviderConfig was not wired
769fd040     fixes BYOK provider wiring (#240)
35640378     regresses BYOK provider-only model IDs by adding startup --model (#263)
origin/main  still fails

A tiny direct Copilot SDK test against copilot-cli 1.0.49 confirms the distinction:

• SessionConfig{Model: "minimax-m2.7"} without Provider fails with Model "minimax-m2.7" is not available.
• SessionConfig{Model: "minimax-m2.7", Provider: &copilot.ProviderConfig{Type: "openai", BaseURL: "...", APIKey: "..."}} succeeds.

So custom providers work through the SDK when the provider config is part of session.create; the failure happens because the CLI startup --model path validates the provider-only model ID against the Copilot catalog first.

Suggested fix

Do not pass --model in ClientOptions.CLIArgs when a custom provider is configured. Keep passing SessionConfig.Model and SessionConfig.Provider on create/resume.

Minimal shape:

provider := providerFromEnv()
cliArgs := modelCLIArgs(defaultModelID, provider.enabled())

func modelCLIArgs(defaultModelID string, customProvider bool) []string {
    if defaultModelID == "" || customProvider {
        return []string{}
    }
    return []string{"--model", defaultModelID}
}

I tested this locally on origin/main; with that change the same eval succeeds and the usage summary correctly reports:

Provider: custom (api.example.com)

This preserves #263's protection for normal GitHub Copilot catalog models while avoiding pre-session catalog validation for BYOK provider model IDs.