feat: pluggable embedding provider — E5 adapter for self-hosted embeddings#150
feat: pluggable embedding provider — E5 adapter for self-hosted embeddings#150SZabolotnii wants to merge 2 commits into
Conversation
…dings Add support for self-hosted E5 embedding models (intfloat/multilingual-e5-small) as an alternative to OpenAI. Enables zero-cost embeddings using existing infrastructure (e.g., Ayona's E5 container on VPS). - New provider router in embedding.ts (GBRAIN_EMBEDDING_PROVIDER=e5|openai) - E5 HTTP adapter (embedding-e5.ts) with auto-dimension detection, retry, batch - Extracted OpenAI provider into embedding-openai.ts (no behavior change) - Dynamic vector(N) in schema — 384d for E5, 1536d for OpenAI - Schema dimension substitution in db.ts, postgres-engine.ts, pglite-engine.ts Deployed and verified on VPS: gbrain init with E5 creates vector(384) schema, doctor reports health_score=90, all checks OK. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12 unit tests for E5 adapter (fetch mock, batching, truncation, dimension detection), OpenAI adapter exports, provider router, and schema dimension substitution. CHANGELOG v0.10.2 entry. CLAUDE.md updated with new files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Thanks for this contribution — and apologies for the slow triage. We did a full pass over the entire PR backlog. gbrain has moved fast, and the maintainer's larger "cathedral" rewrites have superseded a big share of community PRs: the AI gateway + recipes + user_provided_models system replaced almost all individual provider PRs; #1805 fixed the whole Postgres module-singleton class; #1542 unified the type taxonomy; #1657 the retrieval path; #1802 the doctor; and so on. We're closing this one in that cleanup — either the fix already landed on master, it duplicates another PR or merged change, or it's outside the current merge bar. Where a closed PR carried a genuinely valuable idea, we've recorded it in docs/designs/COMMUNITY_IDEAS.md so nothing good is lost (a few may graduate into TODOs). Please don't read the close as a judgment of the work — thank you for contributing. If you believe the underlying issue is still live on the latest master, reopen with a quick note and we'll take another look. 🙏 |
Summary
GBRAIN_EMBEDDING_PROVIDER=e5|openaiselects between self-hosted E5 and OpenAI. OpenAI remains default.embedding-e5.ts): talks to self-hostedintfloat/multilingual-e5-small(or compatible) endpoint with auto-dimension detection, retry, batch splitting.embedding-openai.ts): same behavior, clean separation.vector(384)for E5 orvector(1536)for OpenAI across PGLite and Postgres engines.gbrain initcreatesvector(384)schema, doctor reportshealth_score=90.Env vars
GBRAIN_EMBEDDING_PROVIDERopenaie5oropenaiGBRAIN_E5_URLhttp://embeddings-e5:8000/embedGBRAIN_E5_BATCH_SIZE16Test plan
bun test— 858 pass, 0 failtest/embedding.test.ts— 12 tests covering all providers🤖 Generated with Claude Code