-
Notifications
You must be signed in to change notification settings - Fork 0
feat: embedding fine-tuning pipeline -- wire checkpoint lookup + CLI command #966
Copy link
Copy link
Closed
Labels
prio:mediumShould do, but not blockingShould do, but not blockingscope:medium1-3 days of work1-3 days of workspec:memoryDESIGN_SPEC Section 7 - Memory & PersistenceDESIGN_SPEC Section 7 - Memory & Persistencetype:featureNew feature implementationNew feature implementationv0.6Minor version v0.6Minor version v0.6
Description
Context
PR #962 designed the domain-specific embedding fine-tuning pipeline and shipped EmbeddingFineTuneConfig as a config stub. The config validates its invariants (required fields, path safety) but the Mem0 adapter does not consume it -- fine_tune.enabled=True raises "not yet supported" via a fail-fast validator.
This issue implements the actual pipeline.
Requirements
1. Wire checkpoint lookup into Mem0 adapter
When fine_tune.enabled=True and a valid checkpoint_path exists:
- Load the fine-tuned model instead of the base model at backend initialization
- If checkpoint doesn't exist at the path, log warning and fall back to base model
- Remove the fail-fast validator on
Mem0EmbedderConfig._reject_unimplemented_fine_tune
2. Fine-tuning CLI command
Implement the 4-stage offline pipeline as a CLI command (or Python script):
- Synthetic data generation -- LLM generates query-document pairs from org documents
- Hard negative mining -- base model embeds all passages, selects top-k confusing negatives
- Contrastive fine-tuning -- biencoder training with InfoNCE loss (tau=0.02, 3 epochs, lr=1e-5)
- Deploy -- save checkpoint, update config to point to fine-tuned model
Pipeline design is documented in docs/reference/embedding-evaluation.md.
3. Update docs
- Remove "not yet implemented" notes from
EmbeddingFineTuneConfigdocstring, design spec, and reference page - Update docs to reflect actual behavior instead of planned behavior
Acceptance Criteria
-
fine_tune.enabled=Truewith valid checkpoint loads fine-tuned model - Missing checkpoint falls back to base model with logged warning
- Fail-fast validator removed
- CLI command runs the 4-stage pipeline end-to-end
- Pipeline works on single GPU with no manual annotation
- Tests cover checkpoint lookup (found, not found, disabled)
- Docs updated to reflect implemented behavior
References
- PR docs: LMEB embedding evaluation + CSP accepted risk (#695, #925) #962 --
EmbeddingFineTuneConfig, pipeline design, LMEB evaluation src/synthorg/memory/backends/mem0/config.py-- config model + fail-fast guarddocs/reference/embedding-evaluation.md-- pipeline design and expected improvements- NVIDIA domain-specific fine-tuning blog
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
prio:mediumShould do, but not blockingShould do, but not blockingscope:medium1-3 days of work1-3 days of workspec:memoryDESIGN_SPEC Section 7 - Memory & PersistenceDESIGN_SPEC Section 7 - Memory & Persistencetype:featureNew feature implementationNew feature implementationv0.6Minor version v0.6Minor version v0.6