Skip to content

Add Native Gemini API Compatibility to Server #5

@marksverdhei

Description

@marksverdhei

Add Native Gemini API Compatibility to Server

Overview

Currently, llama-server provides an OpenAI-compatible API (/v1/chat/completions) and natively supports Anthropic's message schema via an inline request/response translator built into the server (convert_anthropic_to_oai).

We want to add native support for the Google Gemini API (e.g., /v1beta/models/...:generateContent or similar endpoints) using the exact same request/response translation pattern that was used for the Anthropic implementation.

Reference Implementations

  • LiteLLM: See how LiteLLM intercepts Gemini API requests and translates them on-the-fly to OpenAI formats: LiteLLM Gemini Handler
  • Anthropic Implementation in llama.cpp: See tools/server/server-common.cpp (convert_anthropic_to_oai) and tools/server/server-task.cpp (to_json_anthropic) in our existing codebase for reference on how to inline the translation.

Checklist

  • Add convert_gemini_to_oai() in tools/server/server-common.cpp to map Gemini's contents, parts, and text into OpenAI's messages and content.
  • Define TASK_RESPONSE_TYPE_GEMINI in tools/server/server-task.h (similar to TASK_RESPONSE_TYPE_ANTHROPIC).
  • Implement to_json_gemini() in tools/server/server-task.cpp to emit Google-formatted responses (candidates, content, parts).
  • Implement format_gemini_sse() in tools/server/server-context.cpp for streaming response support via SSE.
  • Update handle_completions_impl and related logic in server-context.cpp to process Gemini endpoints.
  • Register new Gemini HTTP endpoints (e.g., ctx_http.post("/v1beta/models/:model:generateContent", ...)) in tools/server/server.cpp.
  • Add unit tests in tools/server/tests/unit/ (e.g., test_compat_gemini.py) verifying correct request translation and response wrapping.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions