Skip to content

[Bug]: google-vertex provider returns 404 from Google when models are accessed via openclaw agent, but direct curl with same URL + ADC token works #84804

@devinallen-07

Description

@devinallen-07

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

openclaw agent calls to google-vertex/* models fail with FailoverError: 404 even though the same URL + ADC token return HTTP 200 via curl, and patched debug logs in the google-vertex transport never trigger — indicating the request is being routed through an unexpected code path.

Steps to reproduce

Here's a clean copy-paste version for the GitHub issue:


Steps to Reproduce

Environment: OpenClaw 2026.5.18, macOS, GCP project with aiplatform.googleapis.com API enabled, ADC configured (gcloud auth application-default login).

1. Configure google-vertex provider in openclaw.json:

cat > /tmp/gv.json5 << 'EOF'
{
  "models": {
    "providers": {
      "google-vertex": {
        "baseUrl": "https://aiplatform.googleapis.com",
        "apiKey": "***",
        "models": [
          {
            "id": "gemini-2.5-flash",
            "name": "Gemini 2.5 Flash",
            "reasoning": true,
            "input": ["text", "image"],
            "cost": { "input": 0.30, "output": 2.50, "cacheRead": 0.03, "cacheWrite": 0.30 },
            "contextWindow": 1048576,
            "maxTokens": 65536
          }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "models": { "google-vertex/gemini-2.5-flash": {} }
    }
  },
  "env": {
    "vars": {
      "GOOGLE_CLOUD_PROJECT": "<your-gcp-project>",
      "GOOGLE_CLOUD_LOCATION": "global"
    }
  }
}
EOF
openclaw config patch --file /tmp/gv.json5 --replace-path "models.providers.google-vertex"
openclaw gateway restart

2. Verify model is registered:

openclaw models list | grep gemini-2.5-flash
# google-vertex/gemini-2.5-flash    text+image  1024k  no  yes  configured

3. Call the model:

openclaw agent --agent main --model google-vertex/gemini-2.5-flash --message "hi"

Actual result:

GatewayClientRequestError: FailoverError: 404 <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <title>Error 404 (Not Found)!!1</title>

Expected result: Model response (similar to anthropic-vertex/* models which work correctly with the same ADC).

4. Confirm the URL itself works via direct curl (with the same ADC token):

TOKEN=*** auth application-default print-access-token)
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  "https://aiplatform.googleapis.com/v1/projects/<your-project>/locations/global/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
# HTTP 200

This is the same URL OpenClaw's buildGoogleVertexRequestUrl() in dist/transport-stream-BGlgI5ge.js should construct. So the URL/auth/payload work — but the request from openclaw agent produces a 404 with HTML body (Google's default not-found page), suggesting the actual outgoing request goes to a different URL than expected.

Additional debug: Adding console.log statements at the entry of createGoogleTransportStreamFn (transport-stream.js) and inside createStreamFn (provider-registration.js, the Google plugin) — neither log line is reached when calling a google-vertex/* model. This suggests the request is being routed through a different code path entirely.

Expected behavior

openclaw agent --agent main --model google-vertex/gemini-2.5-flash --message "hi" should return the model's text response and route through createGoogleVertexTransportStreamFn in dist/transport-stream-BGlgI5ge.js — matching the behavior of anthropic-vertex/claude-opus-4-7 calls, which succeed against https://aiplatform.us.rep.googleapis.com using the same ADC credentials in this OpenClaw 2026.5.18 install.

Actual behavior

Calls to any google-vertex/* model immediately return GatewayClientRequestError: FailoverError: 404 <!DOCTYPE html>... <title>Error 404 (Not Found)!!1</title> (Google's default not-found HTML page), while identical requests to anthropic-vertex/* models succeed and the same target URL (https://aiplatform.googleapis.com/v1/projects/<id>/locations/global/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse) returns HTTP 200 when called directly via curl with the gateway's ADC token; additionally, console.log statements added at the entry of createGoogleTransportStreamFn (dist/transport-stream-BGlgI5ge.js) and inside the Google plugin's createStreamFn (dist/provider-registration-DWpi0i_i.js) never appear in the gateway log when a google-vertex/* model is invoked, indicating the request never reaches the expected transport function.

OpenClaw version

OpenClaw 2026.5.18

Operating system

macOS 26.3.1

Install method

npm global

Model

google-vertex/gemini-2.5-flash (also reproduced with google-vertex/gemini-3.5-flash, google-vertex/gemini-3.1-pro-preview, google-vertex/gemini-2.5-pro, and google-vertex/gemini-2.5-flash-lite — all return the same 404).

Provider / routing chain

Direct: OpenClaw gateway → google plugin (dist/extensions/google/index.js) → google-vertex transport (expected: createGoogleVertexTransportStreamFn in dist/transport-stream-BGlgI5ge.js) → Vertex AI endpoint https://aiplatform.googleapis.com/v1/projects/<id>/locations/global/publishers/google/models/<model>:streamGenerateContent?alt=sse, with auth via ADC (gcp-vertex-credentials marker → resolveGoogleVertexAuthorizedUserHeaders minting a Bearer token from ~/.config/gcloud/application_default_credentials.json). No external proxies, gateways, or routers; the gateway is bound to loopback (ws://127.0.0.1:18789).

Additional provider/model setup details

  • Same ADC works for anthropic-vertex (Claude Opus 4.7/4.6, Sonnet 4.6, Haiku 4.5 all succeed via the @openclaw/anthropic-vertex-provider plugin at ~/.openclaw/npm/node_modules/@openclaw/anthropic-vertex-provider), so credentials and ADC discovery are healthy.

  • google provider with API key works for the same Gemini model IDs (google/gemini-2.5-flash, google/gemini-3.5-flash, etc.) using GEMINI_API_KEY from macOS Keychain — confirming the models themselves are reachable from this machine and project.

  • Schema enforces models.providers.google-vertex.baseUrl as required, even though the plugin's providerEndpoints config in dist/extensions/google/openclaw.plugin.json defines host-based detection for both aiplatform.googleapis.com (with googleVertexRegion: global) and *-aiplatform.googleapis.com (with googleVertexRegionHostSuffix).

  • Tried both endpoint patterns, all failed identically:

    • Regional: baseUrl=https://us-central1-aiplatform.googleapis.com + GOOGLE_CLOUD_LOCATION=us-central1
    • Global: baseUrl=https://aiplatform.googleapis.com + GOOGLE_CLOUD_LOCATION=global
  • env.vars in openclaw.json does not propagate to gateway process.env — confirmed via ps eww on the gateway PID showing GOOGLE_CLOUD_PROJECT / GOOGLE_CLOUD_LOCATION absent until I added them to the LaunchAgent's service-env wrapper file. This may be relevant since resolveGoogleVertexProject() reads from process.env directly.

  • Plugin registration confirmed: openclaw plugins list shows google plugin enabled (stock bundle dist/extensions/google/index.js, v2026.5.18), and openclaw models list shows all google-vertex/* entries with Auth: yes and configured tag.

  • openclaw config validate passes for the offending config.

  • Plist StandardErrorPath is /dev/null — debug console.error is dropped, so debug logs were added as console.log (which does reach ~/Library/Logs/openclaw/gateway.log) and still never fired for google-vertex/* calls.

Logs, screenshots, and evidence

**Gateway log — the actual 404 error** (`~/Library/Logs/openclaw/gateway.log`):

2026-05-20T15:40:10.374-06:00 [ws] ⇄ res ✗ agent errorCode=UNAVAILABLE 
  errorMessage=FailoverError: 404 <!DOCTYPE html>
  <html lang=en>
    <meta charset=utf-8>
    <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
    <title>Error 404 (Not Found)!!1</title>
    <style>
      *{margin:0;padding:0}html...
  runId=f9d34afc-89c2-4cee-ab89-4ff246f110a1


**Side-by-side: same URL via curl returns 200, via OpenClaw returns 404:**

$ TOKEN=*** auth application-default print-access-token)
$ for url in \
    "https://aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/us-central1/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse" \
    "https://aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/global/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse" \
    "https://us-central1-aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/us-central1/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse"; do
    curl -s -o /dev/null -w "HTTP %{http_code}  $url\n" "$url" \
      -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
      -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
  done

HTTP 200  https://aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/us-central1/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse
HTTP 200  https://aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/global/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse
HTTP 200  https://us-central1-aiplatform.googleapis.com/v1/projects/<REDACTED>/locations/us-central1/publishers/google/models/gemini-2.5-flash:streamGenerateContent?alt=sse

$ openclaw agent --agent main --model google-vertex/gemini-2.5-flash --message "hi" 2>&1 | tail -3
GatewayClientRequestError: FailoverError: 404 <!DOCTYPE html>
<html lang=en>
  <title>Error 404 (Not Found)!!1</title>


**`openclaw models list` — proving the model is configured and "Auth: yes":**

Model                                      Input      Ctx         Local Auth  Tags
anthropic-vertex/claude-opus-4-7           text+image 977k        no    yes   default,configured
anthropic-vertex/claude-haiku-4-5@20251001 text+image 195k        no    yes   configured
google-vertex/gemini-2.5-pro               text+image 1024k       no    yes   configured
google-vertex/gemini-2.5-flash             text+image 1024k       no    yes   configured
google-vertex/gemini-3.5-flash             text+image 1024k       no    yes   configured
google-vertex/gemini-3.1-pro-preview       text+image 1024k       no    yes   configured
google/gemini-2.5-flash                    text+image 1024k       no    yes   configured

(`anthropic-vertex/*` and `google/*` work; `google-vertex/*` 404s.)

**Debug patches that never triggered** — added to `/opt/homebrew/lib/node_modules/openclaw/dist/`:

// transport-stream-BGlgI5ge.js, entry of createGoogleTransportStreamFn:
console.log("[DEBUG-TRANSPORT-CALLED]", kind, "model.id=", rawModel?.id,
            "model.api=", rawModel?.api, "model.provider=", rawModel?.provider);

// provider-registration-DWpi0i_i.js, top of createStreamFn:
console.log("[DEBUG-CREATESTREAMFN]", "model.id=", model?.id, "model.api=", model?.api,
            "model.provider=", model?.provider, "hasADC=", hasGoogleVertexAuthorizedUserAdcSync());

Neither line ever appears in `~/Library/Logs/openclaw/gateway.log` after restart and multiple `google-vertex/*` calls. The same calls **do** produce the 404 errors logged above, so the gateway is making *some* outbound request — just not via this code path.

**Versions:**

$ openclaw --version
OpenClaw 2026.5.18 (50a2481)

$ node --version
v26.0.0

$ sw_vers
ProductName:    macOS
ProductVersion: 26.3.1
BuildVersion:   arm64

$ openclaw plugins list | grep google
@openclaw/google  google  openclaw  enabled  stock:google/index.js  2026.5.18


**Workaround in use:** `google` provider with `GEMINI_API_KEY` (works fine). The `google-vertex` config is left in place for when this is fixed.

Full investigation notes (with all debug steps) available at `~/.openclaw/workspace/issues/openclaw-google-vertex-404.md` — happy to attach or share if useful.

Impact and severity

  • Affected users/systems/channels: Any OpenClaw user trying to access Google Gemini models via the google-vertex provider (ADC/Vertex AI billing path). The google provider (API key / AI Studio billing) and anthropic-vertex provider are unaffected.
  • Severity: Blocks the Vertex AI routing workflow entirely for Gemini. Workaround exists (use google provider with GEMINI_API_KEY), so not a hard block — but it forces a billing-surface choice users may not want (AI Studio billing vs. unified GCP billing).
  • Frequency: Always reproducible. Every openclaw agent call against any google-vertex/* model fails with the same 404 in this install.
  • Consequence: (1) Cannot route Gemini calls through Vertex AI for unified GCP billing/quota/audit alongside Anthropic on Vertex. (2) Cost-tracking and quota dashboards in GCP miss the Gemini portion of agent traffic. (3) Engineering time lost debugging (in this case ~6 hours across multiple gateway restarts before identifying the workaround). (4) Forces dual-billing setup (GCP for Anthropic-Vertex, Google AI Studio for Gemini API key) which complicates expense management for orgs that prefer single-vendor billing.

Additional information

NOT_ENOUGH_INFO

(We don't have a known-good version for google-vertex on this install — it's been broken since we first configured it today. No prior baseline exists to identify when it regressed, if ever.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.bugSomething isn't workingbug:behaviorIncorrect behavior without a crashclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions