Summary

Fixes two related memory-enforcer bugs that intermittently 500 /v1/embeddings and /v1/rerank with:

RuntimeError: Engine not started. Call start() first.

Both stem from the same gap: eviction is gated only by has_active_requests(), which is blind during the window between acquiring an engine and registering activity on it.

Fixes #1667. Closely related to #1595 (this PR also defuses it).

Bug A — acquire-vs-use eviction race (#1667)

EnginePool.get_engine() returns the engine handle, then releases the pool lock. The request handler then runs outside the lock (engine = await get_embedding_engine(...) ... later await engine.embed(...)). For the non-streaming engines (embedding/reranker), the in-use signal (_active_count via _begin_activity) is registered only after the if self._model is None: raise RuntimeError("Engine not started...") guard inside embed()/rerank(). So has_active_requests() returns False for the entire span from acquire until _begin_activity.

Meanwhile the ProcessMemoryEnforcer's ~1s poll takes the same pool lock and _find_lru_victim() / _is_idle_for_prefill_eviction() / check_ttl_expirations() see the engine as idle, so they evict it. The suspended request resumes against a torn-down engine and raises. The embedding engine (smallest, usually oldest last_access) is the usual victim — the recurring cause of "embedder degraded" episodes for downstream consumers.

Bug B — active-request counter leak (#1595)

The immediate-abort eviction stops a non-streaming engine without the normal per-request completion callbacks (_end_activity), so _active_count can be left non-zero — a phantom "busy" count that corrupts the status API and (via has_active_requests()) can make a stale engine look permanently non-evictable.

Fix — atomic in-use lease + teardown reset

Take the in-use signal at acquire time, under the pool lock, so eviction can never tear down an engine a request holds or is acquiring:

• EngineEntry.in_use: int = 0 refcount.
• get_engine(..., _lease: bool = False) increments entry.in_use under self._lock at both return points (already-loaded + post-load).
• release_engine(model_id) decrements under the lock (floored at 0); acquire(model_id) @asynccontextmanager leases on enter and always releases in finally (so a request error can't leak the lease).
• All three eviction paths skip in_use > 0: _find_lru_victim, _is_idle_for_prefill_eviction, check_ttl_expirations (in addition to has_active_requests()).
• server.py threads _lease / _leased_out so the exact pool-resolved model_id (incl. the default-model fallback) is captured, and releases the lease on validation failure. /v1/embeddings and /v1/rerank wrap the real embed()/rerank() call in async with pool.acquire(model_id) (the embeddings lease is taken inside the deferred StreamingResponse coroutine, which is where the engine is actually used). LLM/STT/TTS/STS handlers are unchanged. The non-lease path keeps the original pool.get_engine(model_id) call contract.
• Memory enforcer eviction path leaks active_requests counter #1595: BaseNonStreamingEngine._reset_activity_tracking() zeros _active_count and clears _activities under _active_lock, called (getattr-guarded) from EnginePool._unload_engine() after stop().

This closes both sub-windows (acquire→activity, and the guard→increment gap inside embed()) and preserves eviction of genuinely idle engines — only engines that are leased/in-use are protected.

Testing

Tests run with the repo's pytest suites (Python 3.11, mlx available):

• tests/test_engine_pool.py + tests/test_embedding.py: 154 passed (incl. 10 new regression tests).
• New regression tests (TestEnginePoolInUseLease, TestResetActivityTracking): the three eviction paths skip in_use > 0; acquire() leases then releases on success and on exception; release_engine floors at 0; _reset_activity_tracking clears a leaked counter; _unload_engine invokes the reset on teardown. Verified these fail against the pre-fix code and pass with the fix (non-vacuous).
• Server/rerank/embeddings suites (tests/test_server.py, tests/test_rerank_models.py, integration endpoint tests): green. The integration MockEnginePool was updated to mirror the new lease API (_lease kwarg + no-op release_engine).
• Full default suite (-m "not slow and not integration" plus the explicit integration endpoint tests): green except two pre-existing test_install_detection.py failures that are environment artifacts of running under a Homebrew-installed interpreter (is_homebrew() detects the Cellar path); they fail identically on unmodified main and are unrelated to this change.

🤖 Generated with Claude Code