-
Notifications
You must be signed in to change notification settings - Fork 0
feat: auto-select embedding model from provider using LMEB rankings + configurable overrides #965
Description
Context
PR #962 established LMEB as the evaluation framework for embedding model selection and documented three deployment tiers with specific model recommendations. However, none of this research flows into the actual user experience -- users currently must manually specify Mem0EmbedderConfig(provider=..., model=..., dims=...) in code.
A normal user will install SynthOrg, connect an Ollama instance with a local GPU or a cloud provider API key, and expect memory retrieval to just work. They should never need to read a reference page to get a good embedding model.
Requirements
1. Auto-selection from connected provider
When a provider is connected (setup wizard or config), the system should:
- Query available embedding models from the provider (leverage existing provider auto-discovery in
providers/) - Rank available models using LMEB scores (the rankings from
docs/reference/embedding-evaluation.md) - Auto-select the best available model for the deployment context
- Infer deployment tier from provider type: Ollama with GPU -> Tier 1/2 candidates, cloud provider -> Tier 1 candidates, CPU-only -> Tier 3 candidates
The LMEB research should become selection logic in code, not just documentation.
2. Template-level override
Company templates (in templates/) should be able to specify a preferred embedding model:
memory:
embedder:
provider: "..." # optional override
model: "..." # optional override
dims: ... # required if model is overriddenIf not specified in the template, auto-selection applies.
3. Runtime settings override (advanced)
Once the application is running, users should be able to override the embedding model via the Settings page in the dashboard (advanced section). This is a runtime-editable setting via the existing settings/ infrastructure.
Important: Changing the embedding model after initial deployment requires recreating the Qdrant collection (existing vectors become incompatible). The UI must warn about this and ideally offer a re-embedding migration path.
4. Expose in company YAML config
Add memory.embedder section to the company config schema with sane defaults derived from auto-selection. Users who know what they want can set it directly in YAML.
5. Setup wizard integration
The setup wizard already asks about provider configuration. Add embedding model selection as an auto-recommended step based on the provider/hardware answers. Show the auto-selected model with an "Advanced: change model" toggle.
Acceptance Criteria
- Embedding model is auto-selected when a provider is connected, using LMEB rankings
- Templates can override the embedding model
- Runtime settings page has an advanced toggle for embedding model override
- Company YAML config supports
memory.embedderwith sane defaults - Setup wizard auto-recommends based on provider context
- Changing embedding model post-deployment warns about Qdrant collection incompatibility
-
Mem0EmbedderConfigdefaults are derived from auto-selection, not hardcoded
References
- PR docs: LMEB embedding evaluation + CSP accepted risk (#695, #925) #962 -- LMEB evaluation and
EmbeddingFineTuneConfigstub docs/reference/embedding-evaluation.md-- model rankings and tier recommendationsdocs/design/memory.md-- embedding model selection sectionsrc/synthorg/providers/-- provider auto-discovery infrastructuresrc/synthorg/templates/-- company template systemsrc/synthorg/settings/-- runtime-editable settings