Summary

• Add 10 custom Prometheus metrics for model lifecycle and inference service management (download duration, model status, time-to-ready, GPU queue depth, reconciliation tracking)
• Enable llama.cpp --metrics flag and health probes (startup + liveness + readiness) on inference pods
• Add PodMonitor template for scraping inference pod metrics via Prometheus
• Initialize OpenTelemetry TracerProvider with OTLP gRPC exporter for distributed tracing (opt-in via OTEL_EXPORTER_OTLP_ENDPOINT env var)
• Add lifecycle alert rules: ModelDownloadSlow, InferenceServiceNotReady, GPUQueueBacklog

Related Issues

• Contributes to Additional Grafana dashboards: inference metrics and cost tracking #5 — Provides the custom Prometheus metrics (tokens/sec, latency, queue depth, model status) needed for inference metrics and cost tracking dashboards
• Contributes to test: Improve unit test coverage across packages #89 — Adds comprehensive unit tests for the new internal/metrics package, improving controller test coverage
• Related to Metal agent: health checks, backpressure, and observability #171 — Establishes operator-level observability patterns (metrics, PodMonitor, alert rules) that the metal agent observability work can build on

Changes

New files

• internal/metrics/metrics.go — Custom metrics registered via controller-runtime registry
• internal/metrics/metrics_test.go — Comprehensive tests for all 10 metrics
• charts/llmkube/templates/inference-podmonitor.yaml — PodMonitor for inference pods

Modified files

• internal/controller/model_controller.go — Download duration, status, and reconcile metrics
• internal/controller/inferenceservice_controller.go — Phase tracking, ready duration, --metrics flag, three-probe pattern (startup/liveness/readiness)
• cmd/main.go — OTEL TracerProvider initialization
• config/prometheus/llmkube-alerts.yaml — 3 new lifecycle alerts
• charts/llmkube/values.yaml — Inference PodMonitor configuration
• go.mod — Promote OTel dependencies from indirect to direct

Design Decisions

Startup probe for model loading tolerance

llama.cpp /health returns 503 during model loading and 200 when ready. Large models (30B+) take 5-30 minutes to load. Instead of relying on initialDelaySeconds (which gave only 75s tolerance), we use Kubernetes' three-probe pattern:

• StartupProbe: failureThreshold=180 * periodSeconds=10 = 30 min budget for model loading. Gates liveness/readiness until model is loaded.
• LivenessProbe: Deadlock detection after startup (tight 15s interval)
• ReadinessProbe: Traffic routing after startup (tight 10s interval)

--metrics always enabled

llama.cpp --metrics has negligible overhead (read-only endpoint). The PodMonitor is enabled: false by default, so metrics aren't scraped unless the user opts in.

Test plan

• All existing tests pass (go test ./...)
• New metrics package tests pass (10 tests covering registration, observations, phase transitions, histogram buckets)
• go vet ./... passes with no issues
• Deploy and verify metrics endpoint: curl -k https://llmkube-controller:8443/metrics | grep llmkube_
• Deploy a model and verify llmkube_model_download_duration_seconds is observed
• Verify PodMonitor scrapes inference pod metrics when enabled
• Verify OTEL traces appear in Tempo when OTEL_EXPORTER_OTLP_ENDPOINT is configured
• Verify startup probe tolerates large model loading (>75s)