Problem

When a new memory is extracted (via session.commit() or direct write), _enqueue_semantic_for_parent enqueues a SemanticMsg with recursive=True (default) for the parent directory. The SemanticDagExecutor then re-processes every file in that directory — calling VLM for summaries and the embedding API for vectorization — regardless of whether they have changed.

This means:

• Storing the 1st memory → processes 1 file
• Storing the 100th memory → processes 100 files
• Storing the 500th memory → processes 500 files

Cumulative cost = 1 + 2 + 3 + ... + n = O(n²)

Observed Impact

With ~500 memories, a single day consumed hundreds of thousands of embedding tokens on Volcengine. Combined with rate limiting (429 errors), the retry storm amplified the cost further — 2,000+ 429 errors in one day.

Root Cause (Code References)

1. memory_extractor.py:596 — _enqueue_semantic_for_parent() creates SemanticMsg without setting recursive=False
2. semantic_processor.py:396 — _generate_single_file_summary() has no caching/mtime check; every file is re-summarized on each run
3. semantic_dag.py — _file_summary_task() unconditionally vectorizes every file

Suggested Fix

One or more of:

• Incremental processing: Skip files whose content has not changed (mtime/hash check)
• Set recursive=False for memory extraction semantic messages (only reprocess the immediate parent directory)
• Add a configurable option like semantic.incremental: true in ov.conf

Environment

• OpenViking v0.2.5
• Embedding: Volcengine doubao-embedding-vision-250615
• VLM: GLM-5 (via OpenAI-compatible API)
• ~500 memories across entities/preferences/events