Skip to content

fix(tests): prevent flaky test_validation by preloading SDK error cla…#632

Merged
davincios merged 1 commit into
Tracer-Cloud:mainfrom
hamzzaaamalik:fix/flaky-validation-monkeypatch
Apr 17, 2026
Merged

fix(tests): prevent flaky test_validation by preloading SDK error cla…#632
davincios merged 1 commit into
Tracer-Cloud:mainfrom
hamzzaaamalik:fix/flaky-validation-monkeypatch

Conversation

@hamzzaaamalik

Copy link
Copy Markdown
Collaborator

prevent flaky test_validation by preloading SDK error classes

_load_anthropic_client() and _load_openai_client() in app/cli/wizard/
validation.py lazily import their SDKs and set both the client class
(Anthropic/OpenAI) and the matching *AuthError global. They re-import
whenever *AuthError is None, which silently overwrites any monkeypatch
of the client class.

The tests only patched Anthropic/OpenAI (not *AuthError), so on the
first call in a pytest-xdist worker the real SDK was re-injected, the
real API was called using ANTHROPIC_API_KEY/OPENAI_API_KEY leaked in
from the CI environment, and the test failed with 'rejected the API
key'. Pass/fail depended on xdist worker distribution — any PR that
added tests could trigger it.

Add an autouse fixture that pre-populates the *AuthError globals with
the real error classes so the loaders' import branch never fires in
tests. Subsequent monkeypatches of Anthropic/OpenAI are now stable.

@greptile-apps

greptile-apps Bot commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a flaky-test race condition in test_validation.py caused by the or AnthropicAuthError is None branch in _load_anthropic_client() / _load_openai_client(). Because the *AuthError globals started as None, any test that only monkeypatched the client class (Anthropic/OpenAI) would silently trigger a full re-import of the SDK on the first call, overwriting the monkeypatch and leaking a live API call. The autouse fixture pre-sets both *AuthError globals to the real (already-imported) error classes so the loader's import branch is never entered during tests.

Confidence Score: 5/5

Safe to merge — the fix is correct, well-scoped, and addresses the root cause without touching production code.

The autouse fixture correctly short-circuits both lazy loaders' import branches by pre-populating the *AuthError globals before any individual monkeypatch runs. The logic chain holds: autouse sets *AuthError → test patches Anthropic/OpenAI → loader sees no None → condition False → fakes returned, real SDK never re-injected. No P0/P1 findings; all remaining considerations are speculative or out of scope.

No files require special attention.

Important Files Changed

Filename Overview
tests/cli/wizard/test_validation.py Adds an autouse fixture that pre-populates AnthropicAuthError/OpenAIAuthError module globals, preventing lazy-import re-injection from overwriting monkeypatched client classes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[_load_anthropic_client called] --> B{Anthropic is None\nOR AnthropicAuthError is None?}

    subgraph BEFORE["Before fix (flaky)"]
        B1{Anthropic is None?} -- No --> B2{AnthropicAuthError is None?}
        B2 -- Yes --> C1[Re-import real SDK\nOverwrites monkeypatched Anthropic!\nAnthropicAuthError = real class]
        C1 --> D1[Returns real Anthropic → live API call → test fails]
        B2 -- No --> E1[Returns monkeypatched Anthropic ✓]
    end

    subgraph AFTER["After fix (stable)"]
        B3["autouse fixture: monkeypatch sets\nAnthropicAuthError = real class\n(before test body runs)"]
        B3 --> B4[Test body: monkeypatch sets\nAnthropicAuthError = fake_client_class]
        B4 --> B5{Anthropic is None? → No\nAnthropicAuthError is None? → No}
        B5 -- False --> E2[Returns fake client ✓\nNo re-import, no live API call]
    end

    A --> BEFORE
    A --> AFTER
Loading

Reviews (1): Last reviewed commit: "fix(tests): prevent flaky test_validatio..." | Re-trigger Greptile

@davincios davincios merged commit 1ce2776 into Tracer-Cloud:main Apr 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants