Problem

Ollama automatically unloads models from RAM after each request, causing significant response time delays as the model needs to be reloaded for every alt-text generation. This is particularly problematic for active Altbot instances handling frequent requests, where users experience noticeable latency.

Solution

This PR adds a configurable keep-alive parameter that controls how long Ollama models stay loaded in RAM. The feature leverages Ollama's built-in --keepalive flag to maintain models in memory between requests.

Changes

Configuration

• Added ollama_keep_alive config option with three modes:
  • -1: Persistent serving (model stays loaded indefinitely)
  • 0: Immediate unload after each request
  • Duration strings: 5m, 30m, 1h, etc.
• Default: 5m (5 minutes) if not specified

Implementation

• Updated OllamaProvider to include keep-alive state
• Modified all Ollama command invocations to include --keepalive parameter
• When set to -1, model is pre-loaded during startup with user feedback
• Applied consistently across image processing and translation operations

User Experience

• Enhanced setup wizard with interactive keep-alive configuration
• Clear explanation of options and trade-offs
• Recommends persistent mode for active instances

Usage Example

[llm]
provider = "ollama"
ollama_model = "llava-phi3"
ollama_keep_alive = "-1"  # Keep model loaded persistently

Performance Impact

Benefits

1. Dramatically improved response times for active instances (no model loading delays)
2. Configurable to balance performance vs. resource usage
3. Backward compatible with sensible defaults
4. User-friendly setup wizard guides optimal configuration

This feature is ideal for the main Altbot instance and other high-traffic deployments where response time is critical.

Resolves #40

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.