Skip to content

feat: auto-select embedding model from provider using LMEB rankings + configurable overrides #965

@Aureliolo

Description

@Aureliolo

Context

PR #962 established LMEB as the evaluation framework for embedding model selection and documented three deployment tiers with specific model recommendations. However, none of this research flows into the actual user experience -- users currently must manually specify Mem0EmbedderConfig(provider=..., model=..., dims=...) in code.

A normal user will install SynthOrg, connect an Ollama instance with a local GPU or a cloud provider API key, and expect memory retrieval to just work. They should never need to read a reference page to get a good embedding model.

Requirements

1. Auto-selection from connected provider

When a provider is connected (setup wizard or config), the system should:

  • Query available embedding models from the provider (leverage existing provider auto-discovery in providers/)
  • Rank available models using LMEB scores (the rankings from docs/reference/embedding-evaluation.md)
  • Auto-select the best available model for the deployment context
  • Infer deployment tier from provider type: Ollama with GPU -> Tier 1/2 candidates, cloud provider -> Tier 1 candidates, CPU-only -> Tier 3 candidates

The LMEB research should become selection logic in code, not just documentation.

2. Template-level override

Company templates (in templates/) should be able to specify a preferred embedding model:

memory:
  embedder:
    provider: "..."        # optional override
    model: "..."           # optional override
    dims: ...              # required if model is overridden

If not specified in the template, auto-selection applies.

3. Runtime settings override (advanced)

Once the application is running, users should be able to override the embedding model via the Settings page in the dashboard (advanced section). This is a runtime-editable setting via the existing settings/ infrastructure.

Important: Changing the embedding model after initial deployment requires recreating the Qdrant collection (existing vectors become incompatible). The UI must warn about this and ideally offer a re-embedding migration path.

4. Expose in company YAML config

Add memory.embedder section to the company config schema with sane defaults derived from auto-selection. Users who know what they want can set it directly in YAML.

5. Setup wizard integration

The setup wizard already asks about provider configuration. Add embedding model selection as an auto-recommended step based on the provider/hardware answers. Show the auto-selected model with an "Advanced: change model" toggle.

Acceptance Criteria

  • Embedding model is auto-selected when a provider is connected, using LMEB rankings
  • Templates can override the embedding model
  • Runtime settings page has an advanced toggle for embedding model override
  • Company YAML config supports memory.embedder with sane defaults
  • Setup wizard auto-recommends based on provider context
  • Changing embedding model post-deployment warns about Qdrant collection incompatibility
  • Mem0EmbedderConfig defaults are derived from auto-selection, not hardcoded

References

  • PR docs: LMEB embedding evaluation + CSP accepted risk (#695, #925) #962 -- LMEB evaluation and EmbeddingFineTuneConfig stub
  • docs/reference/embedding-evaluation.md -- model rankings and tier recommendations
  • docs/design/memory.md -- embedding model selection section
  • src/synthorg/providers/ -- provider auto-discovery infrastructure
  • src/synthorg/templates/ -- company template system
  • src/synthorg/settings/ -- runtime-editable settings

Metadata

Metadata

Assignees

No one assigned

    Labels

    prio:highImportant, should be prioritizedscope:large3+ days of workspec:memoryDESIGN_SPEC Section 7 - Memory & Persistencetype:featureNew feature implementationv0.6Minor version v0.6

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions