Add tests for horizontal scaling operator behaviors#4388
Conversation
Add SessionStorageConfig to pkg/vmcp/config.Config and populate it in the VirtualMCPServer converter when spec.sessionStorage.provider is "redis". This allows vMCP pods to discover Redis connection parameters (address, db, keyPrefix) at startup via the existing config injection path. The Redis password is excluded from the config type and is injected separately as the THV_SESSION_REDIS_PASSWORD environment variable by the deployment builder. Closes: #4214
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4388 +/- ##
=======================================
Coverage 69.48% 69.49%
=======================================
Files 485 485
Lines 49576 49585 +9
=======================================
+ Hits 34450 34460 +10
+ Misses 12461 12457 -4
- Partials 2665 2668 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds automated coverage around horizontal scaling behaviors in the ToolHive operator, focusing on VirtualMCPServer session storage configuration and Redis password injection, plus integration/e2e coverage for replica behavior and RunConfig scaling fields.
Changes:
- Added VirtualMCPServer e2e scenario coverage for scaling (replicas 1↔2) and SessionStorageWarning condition behavior.
- Added envtest integration coverage for VirtualMCPServer replica handling (explicit replicas + nil/HPA-compat behavior) and MCPServer RunConfig scaling config population.
- Added controller-level unit tests verifying vMCP ConfigMap
sessionStorageYAML content and Redis passwordSecretKeyRefenv var injection.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/thv-operator/virtualmcp/virtualmcpserver_scaling_test.go | New real-cluster e2e coverage for VMCP replica scaling and SessionStorageWarning behavior |
| cmd/thv-operator/test-integration/virtualmcp/virtualmcpserver_replicas_integration_test.go | New envtest integration checks for VMCP replica passthrough + HPA-compat preservation |
| cmd/thv-operator/test-integration/mcp-server/mcpserver_runconfig_integration_test.go | Adds integration assertions for RunConfig ScalingConfig presence/absence based on spec |
| cmd/thv-operator/controllers/virtualmcpserver_vmcpconfig_test.go | Adds unit test for ConfigMap YAML sessionStorage section population/omission |
| cmd/thv-operator/controllers/virtualmcpserver_deployment_test.go | Adds unit test ensuring Redis password is injected via SecretKeyRef env var (no plaintext) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cmd/thv-operator/test-integration/virtualmcp/virtualmcpserver_replicas_integration_test.go
Outdated
Show resolved
Hide resolved
cmd/thv-operator/controllers/virtualmcpserver_deployment_test.go
Outdated
Show resolved
Hide resolved
Unit tests (controllers package): - TestConfigMapContent_SessionStorage: verifies that ensureVmcpConfigConfigMap produces a ConfigMap YAML with the sessionStorage section populated when provider=redis, and absent when nil or memory. - TestDeploymentForVirtualMCPServer_WithRedisPassword: verifies that deploymentForVirtualMCPServer injects THV_SESSION_REDIS_PASSWORD as a SecretKeyRef env var and never as a plaintext value. Integration tests (envtest with real CRDs + controller loop): - MCPServer: ScalingConfig (backend_replicas + session_redis) is written to the RunConfig ConfigMap when backendReplicas and Redis sessionStorage are set; omitted when absent. - VirtualMCPServer: spec.replicas is reflected in Deployment.Spec.Replicas; nil replicas produces a nil Deployment replicas field for HPA compatibility. E2E tests (Kind cluster via thv-operator-e2e-test-run): - VirtualMCPServer created with replicas=2 runs 2 ready pods and sets SessionStorageWarning condition when no Redis is configured. - Scale lifecycle (1→2→1): verifies Deployment replicas update in-place, 2 pods come up, SessionStorageWarning fires on scale-up and clears on scale-down. Closes: #4220 Fix ignored AddToScheme errors in unit tests Replace `_ = *.AddToScheme(scheme)` with `require.NoError(t, *.AddToScheme(scheme))` in virtualmcpserver_deployment_test.go and virtualmcpserver_vmcpconfig_test.go so that scheme registration failures cause an immediate, clear test failure rather than a confusing error later with a partially configured scheme. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ation Annotation changes do not increment metadata.generation on Kubernetes resources. The previous trigger used `vmcp.Annotations["test/trigger-reconcile"] = "true"`, which left generation unchanged, so `triggerGeneration = vmcp.Generation + 1` was always one ahead of ObservedGeneration and the Eventually timed out. Fix: trigger via `vmcp.Spec.ServiceType = "ClusterIP"` (the default value, semantically a no-op) which IS a spec change and DOES increment Generation. After the Update, controller-runtime populates the object in-place with the server response, so `vmcp.Generation` is already the post-increment value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jerm-dro
left a comment
There was a problem hiding this comment.
question: The tests verify that the infrastructure scales correctly (replica counts, pod readiness, SessionStorageWarning conditions), but they don't send any MCP traffic through the scaled-out vMCP. Is this an intentional gap, or is cross-pod request handling planned for a follow-up?
Without a request hitting a different pod than the one that created the session, we're testing that pods exist but not that horizontal scaling actually works end-to-end. A few approaches that would demonstrate "any pod can handle any request":
- Scale to 2 with Redis, create a session via
tools/list, delete the pod that handled it, then sendtools/callon the same session and verify it succeeds on the surviving pod - Scale to 0 and back to 1, then send a request — proves a completely fresh pod with no in-memory state can serve traffic
- Send N requests through the Service and verify responses come back successfully (doesn't prove session affinity, but proves all pods are functional)
No preference on which approach — just looking for something that closes the loop beyond infrastructure assertions.
There was a problem hiding this comment.
Large PR Detected
This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.
How to unblock this PR:
Add a section to your PR description with the following format:
## Large PR Justification
[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformationAlternative:
Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.
See our Contributing Guidelines for more details.
This review will be automatically dismissed once you add the justification section.
Large PR justification has been provided. Thank you!
|
✅ Large PR justification has been provided. The size review has been dismissed and this PR can now proceed with normal review. |
Summary
Adds test coverage for the horizontal scaling operator behaviors introduced in
the companion PRs (THV-0047). The existing unit tests were added incrementally
with each implementation task; this PR closes gaps at three levels:
Unit tests (
cmd/thv-operator/controllers/):TestConfigMapContent_SessionStorage: verifiesensureVmcpConfigConfigMapproduces a ConfigMap YAML with
sessionStoragepopulated whenprovider=redis, and absent when nil or memory.TestDeploymentForVirtualMCPServer_WithRedisPassword: verifiesdeploymentForVirtualMCPServerinjectsTHV_SESSION_REDIS_PASSWORDas aSecretKeyRefenv var and never as a plaintext value.Integration tests (envtest —
cmd/thv-operator/test-integration/):ScalingConfig(backend_replicas+session_redis) is writtento the RunConfig ConfigMap when
backendReplicasand RedissessionStorageare set; omitted when absent.
spec.replicas=3is reflected inDeployment.Spec.Replicas;nil
spec.replicaspreserves an externally-set replica count acrossreconciliations (HPA-compatible contract, gated on
status.observedGeneration).E2E tests (real Kind cluster —
test/e2e/thv-operator/virtualmcp/):virtualmcpserver_scaling_test.go: two ordered contexts covering the fullcluster-level lifecycle:
replicas=2at creation → 2 ready pods,SessionStorageWarningcondition set.
up,
SessionStorageWarningfires on scale-up and clears on scale-down.Fixes #4220
Type of change
Test plan
task test)task operator-test-integration)task thv-operator-e2e-test-run)task lint-fix)Does this introduce a user-facing change?
No.
Large PR Justification
This adds complete testing to the feature referenced. Cannot be split.