Overview: Empirical tests of web fetch behavior across AI agent platforms, in support of the Agent-Friendly Documentation Spec and specifically contributes data to the Known Platform Limits. Full methodology and findings: testing results docs.

Origin: extension of Dachary Carey's Agent Web Fetch Spelunking, which documented Claude Code's web fetch pipeline in detail.

Goal: measure what actually happens between "agent fetches URL" and "model sees content" - truncation limits, HTML processing, content negotiation - for platforms that don't document these details.

Each platform has two tracks:

• Interpreted - ask the model to reflect on what it retrieved; captures self-perception and estimation variance
• Raw - Python extracts measurements directly from the response object; produces citable, reproducible numbers

Cursor and Copilot testing use a hybrid approach: raw track saves verbatim output to disk for programmatic analysis; interpreted track asks the agent to self-report metrics from fetched content. Visit the Cursor Friction Note and Copilot Friction Note for methodology challenges unique to IDE-based testing.

• Python 3.8+
• API keys for the applicable platforms
• Copilot chat if applicable
• Cursor IDE if applicable

# Clone the repo
git clone https://github.com/rhyannonjoy/agent-ecosystem-testing.git
cd agent-ecosystem-testing

# Create and activate a virtual environment
python3 -m venv venv
# On Windows: venv\Scripts\activate
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Set API keys
export ANTHROPIC_API_KEY="key-here"
export GOOGLE_GEMINI_API_KEY="key-here"
export OPEN_AI_API_KEY="key-here"

# Claude-interpreted
python claude-api/web_fetch_test.py
# Raw
python claude-api/web_fetch_test_raw.py

# ChatGPT-interpreted
python open-ai-web-search/web_search_test.py
# Raw
python open-ai-web-search/web_search_test_raw.py

# Copilot-interpreted
python copilot-web-content-retrieval/web_content_retrieval_testing_framework.py --test {test ID} --track interpreted
# Raw
python copilot-web-content-retrieval/web_content_retrieval_testing_framework.py --test {test ID} --track raw

# Cursor-interpreted
python cursor-web-fetch/web_fetch_testing_framework.py --test {test ID} --track interpreted
# Raw
python cursor-web-fetch/web_fetch_testing_framework.py --test {test ID} --track raw

# Gemini-interpreted
python gemini-url-context/url_context_test.py
# Raw
python gemini-url-context/url_context_test_raw.py

Free Tier Limits: Claude API not available on the free-tier, the API is pay-as-you-go; 5 RPM and 20 RPD limits for gemini-2.5-flash - running both tracks in the same day exhaust the daily quota; plan runs across days or use a paid tier; Copilot and Cursor tests available on Pro Plan; OpenAI requires a minimum credit top-up (~$5) before any API call succeeds regardless of model - insufficient_quota is an account-level block, not a rate limit; set RATE_LIMIT_SLEEP_SECONDS = 0 in the scripts if you're on a paid tier

Both tracks test the same four URLs. The Claude-interpreted track asks the model to describe what it received; the raw track extracts character counts, CSS indicators, and truncation signals programmatically from the web_fetch_tool_result block.

Unlike API-based platforms, Copilot testing uses manual chat sessions in the VS Code IDE. The framework generates prompts, but execution requires copy-paste into the Copilot chat window. Both tracks test 11 distinct URLs across baseline, structured content, offset, and edge case categories. The primary finding is that Copilot autonomously selects between two retrieval mechanisms -fetch_webpage and curl - with no prompt control, and the mechanism selected determines output format more than any other variable.

Copilot has no publicly documented web fetch mechanism. fetch_webpage surfaced through tool logs. When curl substitution occurs, it happens without disclosure in the chat UI. Read the Friction Note for analysis.

Unlike testing platforms with API web fetch tools, Cursor testing uses manual chat sessions with the Cursor IDE. The framework generates prompts, but execution is intentionally not automated and requires copy-paste into the Cursor IDE. Interpreted track asks Cursor to self-report; raw track saves outputs to disk for measurement. Both tracks test 13 distinct URLs. Tests focus on truncation behavior, backend routing variance, content conversion patterns, and reproducibility.

Both tracks test the same eight URLs and content types. The Gemini-interpreted track asks the model to characterize each retrieval; the raw track reads url_retrieval_status enums and tool_use_prompt_token_count directly from the response object.

Two tracks test the same queries through different API surfaces. The ChatGPT-interpreted track uses the Chat Completions API with gpt-4o-mini-search-preview - search is always implicit. The raw track uses the Responses API with gpt-4o + web_search_preview - tool invocation is conditional and explicitly observable.

If you run these tests and get results, please open an issue or PR with your findings.

Especially useful: Run the same tests from different network locations or at different times to check for caching behavior vs live-fetch variation; run PDF tests against different source URLs to distinguish server-blocking from tool limitations; run Cursor tests with explicit model selection to measure model-specific variance; test Cursor's MCP server integration vs native web fetch behavior