The <Inference> effect is currently host-backed only — there is no way to write a user-defined handle[Inference] block in Vera code. This prevents local mocking, replay testing, and routing to local models without going through the host.
Use cases
- Testing: mock
Inference.complete in unit tests without a real API key
- Local models: route to llama.cpp, Ollama, or any OpenAI-compatible endpoint via a custom handler
- Replay: record/replay LLM responses for deterministic tests
Design note
User-defined effect handlers require the handler dispatch mechanism (currently only wired for user-declared effects). The built-in Inference effect would need to become handleable like Http and IO. This is the same limitation as the Http effect.
Related
Deferred from #61 (v0.0.101).
The
<Inference>effect is currently host-backed only — there is no way to write a user-definedhandle[Inference]block in Vera code. This prevents local mocking, replay testing, and routing to local models without going through the host.Use cases
Inference.completein unit tests without a real API keyDesign note
User-defined effect handlers require the handler dispatch mechanism (currently only wired for user-declared effects). The built-in Inference effect would need to become handleable like
HttpandIO. This is the same limitation as the Http effect.Related
Deferred from #61 (v0.0.101).