[codex] finish voyage embedding slice#197
Closed
trymhaak wants to merge 1 commit into
Closed
Conversation
647530b to
bef0045
Compare
aloysiusmartis
pushed a commit
to aloysiusmartis/gbrain
that referenced
this pull request
Apr 18, 2026
Adds a provider-agnostic EmbeddingProvider interface so gbrain can use Gemini (text-embedding-004/gemini-embedding-001) instead of OpenAI, selected via GBRAIN_EMBEDDING_PROVIDER env var. The public embed/embedBatch API in embedding.ts is unchanged — callers see no diff. Architecture: - src/core/embedding-provider.ts — EmbeddingProvider interface, factory (getActiveProvider), isEmbeddingAvailable(), resetActiveProvider() - src/core/providers/openai-embedder.ts — OpenAI impl extracted from embedding.ts - src/core/providers/gemini-embedder.ts — Gemini impl with Matryoshka dims - src/core/providers/retry-utils.ts — shared exponentialDelay + sleep Critical fix: operations.ts put_page had hardcoded !process.env.OPENAI_API_KEY, so Gemini users got silent no-embed on every import. Replaced with isEmbeddingAvailable() which checks whichever provider is active. New command: gbrain migrate --provider openai|gemini [--dimensions N] - ALTER TABLE (only when dims change) - Re-embeds all chunks with the new provider - Updates config table + config.json - Remote guard: CLI-only, cannot be called via MCP Schema: getPGLiteSchema(dims, model) replaces hardcoded vector(1536) in PGLite DDL so new Gemini brains get vector(768) from init. Config: GBrainConfig gains embedding_provider + embedding_dimensions; loadConfig() propagates them to env on startup (does not override if already set). Init: gbrain init --provider gemini [--dimensions N] wires provider at brain creation time. Usage: GBRAIN_EMBEDDING_PROVIDER=gemini gbrain init # Gemini brain, 768 dims gbrain migrate --provider gemini # migrate existing brain gbrain migrate --provider openai # migrate back Relates to: upstream PR garrytan#197 (voyage embedding) — same territory but this approach uses an interface/factory pattern that supports N providers without modifying the call sites each time. Co-authored-by: Al's bot <amartis@celitotech.com>
6 tasks
aloysiusmartis
added a commit
to aloysiusmartis/gbrain
that referenced
this pull request
Apr 18, 2026
Adds a provider-agnostic EmbeddingProvider interface so gbrain can use Gemini (text-embedding-004/gemini-embedding-001) instead of OpenAI, selected via GBRAIN_EMBEDDING_PROVIDER env var. The public embed/embedBatch API in embedding.ts is unchanged — callers see no diff. Architecture: - src/core/embedding-provider.ts — EmbeddingProvider interface, factory (getActiveProvider), isEmbeddingAvailable(), resetActiveProvider() - src/core/providers/openai-embedder.ts — OpenAI impl extracted from embedding.ts - src/core/providers/gemini-embedder.ts — Gemini impl with Matryoshka dims - src/core/providers/retry-utils.ts — shared exponentialDelay + sleep Critical fix: operations.ts put_page had hardcoded !process.env.OPENAI_API_KEY, so Gemini users got silent no-embed on every import. Replaced with isEmbeddingAvailable() which checks whichever provider is active. New command: gbrain migrate --provider openai|gemini [--dimensions N] - ALTER TABLE (only when dims change) - Re-embeds all chunks with the new provider - Updates config table + config.json - Remote guard: CLI-only, cannot be called via MCP Schema: getPGLiteSchema(dims, model) replaces hardcoded vector(1536) in PGLite DDL so new Gemini brains get vector(768) from init. Config: GBrainConfig gains embedding_provider + embedding_dimensions; loadConfig() propagates them to env on startup (does not override if already set). Init: gbrain init --provider gemini [--dimensions N] wires provider at brain creation time. Usage: GBRAIN_EMBEDDING_PROVIDER=gemini gbrain init # Gemini brain, 768 dims gbrain migrate --provider gemini # migrate existing brain gbrain migrate --provider openai # migrate back Relates to: upstream PR garrytan#197 (voyage embedding) — same territory but this approach uses an interface/factory pattern that supports N providers without modifying the call sites each time. Co-authored-by: Al's bot <aloysiusmartis@users.noreply.github.com>
Author
|
Closing this: continuing the work on the fork instead of upstream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
This finishes the local Voyage embedding slice and turns it into a deliberate, test-backed change set instead of a half-finished dirty diff.
embeddingandmodelmetadata whengbrain embed <slug>orgbrain embed --stalehits pages with a mix of fresh and stale chunkssrc/core/embedding.tsviaresolveEmbeddingConfig()VOYAGE_API_KEYis present while keeping OpenAI as the default pathEMBEDDING_MODELon chunks created during import embeddingembeddingsEnabled()logic in hybrid search instead of hard-codingOPENAI_API_KEYWhy it changed
The branch had two real issues:
This closes both gaps.
Impact
gbrain embed --staleis now safe on partially embedded pages.Root cause
The stale-embed updater rebuilt chunk rows from the newly embedded subset and used
undefinedfor missing entries, which wiped pre-existing metadata on untouched chunks. Separately, the Voyage-provider path had been added as a dirty local slice without direct regression tests on provider selection or import-time model stamping.Validation
bun testbun build --compile --outfile /tmp/gbrain-current-build src/cli.ts