Description

Extend deploymentForVirtualMCPServer in virtualmcpserver_deployment.go to support horizontal scaling for VirtualMCPServer pods. This task sets Deployment.Spec.Replicas from spec.replicas with nil-passthrough (for HPA compatibility), adds terminationGracePeriodSeconds to the pod spec for graceful shutdown, and injects the Redis password as an environment variable (THV_SESSION_REDIS_PASSWORD) from SessionStorageConfig.PasswordRef using the established SecretKeyRef env var pattern.

Context

Epic THV-0047 introduces horizontal scaling support for the ToolHive operator. This task covers the VirtualMCPServer deployment builder changes needed for that support. Once TASK-003 (#275 — regenerate deepcopy and CRD manifests) has made the new Replicas and SessionStorage fields available in the compiled API types, this task consumes them in the deployment builder to make the vMCP Deployment reflect the desired replica count and inject session storage credentials into pods.

The nil-passthrough pattern for replicas is critical: when spec.replicas is nil, Deployment.Spec.Replicas must be omitted (not set to 1), preserving compatibility with HPAs and manual scaling. Redis password injection follows the existing buildOIDCEnvVars and buildHMACSecretEnvVar patterns already in the same file, which source secrets via ValueFrom.SecretKeyRef.

Dependencies: #275 (TASK-003 — Regenerate deepcopy and CRD manifests)
Blocks: TASK-008 (Unit Tests)

Acceptance Criteria

• When spec.replicas is set to a non-nil value (e.g., 3), deploymentForVirtualMCPServer sets Deployment.Spec.Replicas to that value
• When spec.replicas is nil, Deployment.Spec.Replicas is nil (not set to int32(1) or any other default), allowing HPAs and external scaling tools to manage replica counts
• terminationGracePeriodSeconds is set on the pod spec (Deployment.Spec.Template.Spec.TerminationGracePeriodSeconds) in all cases, regardless of replica count
• When spec.sessionStorage is non-nil, spec.sessionStorage.provider == "redis", and spec.sessionStorage.passwordRef is non-nil, buildEnvVarsForVmcp (or a dedicated helper) appends a corev1.EnvVar named THV_SESSION_REDIS_PASSWORD with ValueFrom.SecretKeyRef sourced from passwordRef.name / passwordRef.key
• When spec.sessionStorage is nil, provider != "redis", or passwordRef is nil, no THV_SESSION_REDIS_PASSWORD env var is added (no spurious injection)
• The Redis password env var is injected into the vmcp container in the same manner as VMCP_OIDC_CLIENT_SECRET and VMCP_SESSION_HMAC_SECRET — i.e., via ValueFrom.SecretKeyRef, never as a plaintext Value
• Existing deployment update logic in virtualmcpserver_controller.go continues to preserve Spec.Replicas on update (i.e., when the spec changes, only Spec.Template, Labels, and Annotations are overwritten — not Spec.Replicas). This is already the case; the task must not regress this behavior.
• All existing unit and integration tests for VirtualMCPServerReconciler continue to pass after this change
• Code reviewed and approved

Technical Approach

Recommended Implementation

There are three distinct changes within virtualmcpserver_deployment.go:

1. Nil-passthrough replica setting in deploymentForVirtualMCPServer

Replace the current hardcoded replicas := int32(1) / Replicas: &replicas pattern with spec-driven logic:

// Before (current):
replicas := int32(1)
// ...
Spec: appsv1.DeploymentSpec{
    Replicas: &replicas,
    // ...
}

// After:
// spec.replicas == nil means hands-off (HPA/manual scaling)
// spec.replicas != nil means set the exact value from spec
Spec: appsv1.DeploymentSpec{
    Replicas: vmcp.Spec.Replicas, // nil-passthrough: omits field when nil
    // ...
}

Note that VirtualMCPServerSpec.Replicas is a *int32 (added by TASK-002, available after TASK-003). Directly assigning vmcp.Spec.Replicas to Deployment.Spec.Replicas (both are *int32) achieves the nil-passthrough without any conditional logic.

2. Set terminationGracePeriodSeconds on the pod spec

Add TerminationGracePeriodSeconds to corev1.PodSpec inside deploymentForVirtualMCPServer. The value should be defined as a package-level constant or derived from a spec field. Based on RFC THV-0047, a sensible default (e.g., 30 seconds) should be set unconditionally:

// Add a package-level constant for the grace period
const vmcpTerminationGracePeriodSeconds = int64(30)

// In deploymentForVirtualMCPServer, inside corev1.PodSpec:
Spec: corev1.PodSpec{
    TerminationGracePeriodSeconds: int64Ptr(vmcpTerminationGracePeriodSeconds),
    ServiceAccountName:            serviceAccountName,
    // ...
}

Add a helper int64Ptr if one does not already exist in the package, or reuse an existing one (check mcpserver_controller.go for int32Ptr as a pattern reference).

3. Redis password env var injection

Extend buildEnvVarsForVmcp (or add a dedicated buildRedisEnvVars helper following the buildOIDCEnvVars / buildHMACSecretEnvVar pattern) to inject THV_SESSION_REDIS_PASSWORD when the session storage provider is Redis and a password reference is configured:

// In buildEnvVarsForVmcp, after the existing HMAC secret injection:
env = append(env, r.buildRedisPasswordEnvVar(vmcp)...)

// New helper — follow buildOIDCEnvVars structure exactly:
func (*VirtualMCPServerReconciler) buildRedisPasswordEnvVar(
    vmcp *mcpv1alpha1.VirtualMCPServer,
) []corev1.EnvVar {
    if vmcp.Spec.SessionStorage == nil ||
        vmcp.Spec.SessionStorage.Provider != "redis" ||
        vmcp.Spec.SessionStorage.PasswordRef == nil {
        return nil
    }
    return []corev1.EnvVar{{
        Name: "THV_SESSION_REDIS_PASSWORD",
        ValueFrom: &corev1.EnvVarSource{
            SecretKeyRef: &corev1.SecretKeySelector{
                LocalObjectReference: corev1.LocalObjectReference{
                    Name: vmcp.Spec.SessionStorage.PasswordRef.Name,
                },
                Key: vmcp.Spec.SessionStorage.PasswordRef.Key,
            },
        },
    }}
}

The existing deploymentNeedsUpdate logic in virtualmcpserver_controller.go (lines ~930–937) already preserves Spec.Replicas on update. This task does not need to change that behavior — it only changes what deploymentForVirtualMCPServer returns on initial creation.

Patterns & Frameworks

• Nil-passthrough for *int32 replicas: The architecture mandates that Deployment.Spec.Replicas == nil when spec.replicas is nil. Direct pointer assignment (Replicas: vmcp.Spec.Replicas) achieves this without branching.
• SecretKeyRef env var pattern: Follow buildOIDCEnvVars and buildHMACSecretEnvVar in the same file — corev1.EnvVar with ValueFrom.SecretKeyRef pointing to the referenced secret's name and key. Never use a plaintext Value.
• Helper function decomposition: Keep buildEnvVarsForVmcp as a coordinator that delegates to focused helpers. Add buildRedisPasswordEnvVar as a new private helper in the same style.
• Package-level constants for numeric values: New integer values (grace period seconds) should be declared as typed constants at the top of the file alongside the existing vmcpDefaultPort, vmcpLivenessInitialDelay, etc.

Code Pointers

• cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 112–201 — deploymentForVirtualMCPServer: primary function to modify; replace replicas := int32(1) / Replicas: &replicas with nil-passthrough and add TerminationGracePeriodSeconds to corev1.PodSpec
• cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 256–287 — buildEnvVarsForVmcp: add buildRedisPasswordEnvVar call after the HMAC secret line
• cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 289–329 — buildOIDCEnvVars: reference pattern for SecretKeyRef env var injection; buildRedisPasswordEnvVar should follow this structure
• cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 331–348 — buildHMACSecretEnvVar: second reference pattern; note it uses a constructed secret name rather than a spec-provided one (Redis uses spec-provided PasswordRef)
• cmd/thv-operator/controllers/virtualmcpserver_controller.go lines 926–937 — deployment update logic that preserves Spec.Replicas; must not be modified by this task
• cmd/thv-operator/controllers/mcpserver_controller.go line 934 — replicas := int32(1) — parallel pattern in MCPServer (TASK-004 will change this; verify both tasks use consistent nil-passthrough)
• cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go — VirtualMCPServerSpec: after TASK-003, Replicas *int32 and SessionStorage *SessionStorageConfig are available here
• cmd/thv-operator/api/v1alpha1/mcpserver_types.go — SessionStorageConfig struct definition (defined once, referenced from VirtualMCPServerSpec.SessionStorage)

Component Interfaces

The key interface changes in deploymentForVirtualMCPServer:

// deploymentForVirtualMCPServer — key changes (illustrative, not full function)
func (r *VirtualMCPServerReconciler) deploymentForVirtualMCPServer(
    ctx context.Context,
    vmcp *mcpv1alpha1.VirtualMCPServer,
    vmcpConfigChecksum string,
    typedWorkloads []workloads.TypedWorkload,
) *appsv1.Deployment {
    // REMOVED: replicas := int32(1)
    // ...
    dep := &appsv1.Deployment{
        // ...
        Spec: appsv1.DeploymentSpec{
            Replicas: vmcp.Spec.Replicas, // nil when spec.replicas unset; pointer value when set
            // ...
            Template: corev1.PodTemplateSpec{
                Spec: corev1.PodSpec{
                    TerminationGracePeriodSeconds: int64Ptr(vmcpTerminationGracePeriodSeconds),
                    // ...existing fields...
                },
            },
        },
    }
    // ...
}

New helper function signature:

// buildRedisPasswordEnvVar returns the THV_SESSION_REDIS_PASSWORD env var when
// sessionStorage.provider == "redis" and passwordRef is set; returns nil otherwise.
func (*VirtualMCPServerReconciler) buildRedisPasswordEnvVar(
    vmcp *mcpv1alpha1.VirtualMCPServer,
) []corev1.EnvVar

Testing Strategy

Unit tests for the changes in this task are covered by TASK-008. However, the developer should verify the following during implementation to ensure the changes are correct before TASK-008 formally adds them as test cases:

Unit Tests (to be formally written in TASK-008)

• deploymentForVirtualMCPServer with spec.replicas = nil returns a Deployment with Spec.Replicas == nil
• deploymentForVirtualMCPServer with spec.replicas = 1 returns a Deployment with *Spec.Replicas == 1
• deploymentForVirtualMCPServer with spec.replicas = 3 returns a Deployment with *Spec.Replicas == 3
• deploymentForVirtualMCPServer always returns a Deployment with Spec.Template.Spec.TerminationGracePeriodSeconds set to the expected constant value
• buildRedisPasswordEnvVar with provider == "redis" and non-nil passwordRef returns one corev1.EnvVar named THV_SESSION_REDIS_PASSWORD with correct ValueFrom.SecretKeyRef fields
• buildRedisPasswordEnvVar with nil SessionStorage returns no env vars
• buildRedisPasswordEnvVar with provider == "memory" returns no env vars
• buildRedisPasswordEnvVar with provider == "redis" and nil passwordRef returns no env vars

Integration Tests

• Not in scope for this task; integration tests are covered in TASK-008

Edge Cases

• spec.replicas = 0 — ensure Deployment.Spec.Replicas is set to int32(0) (pointer to zero, not nil), as this is a valid value meaning "scale to zero"
• Verify existing tests in the VirtualMCPServer controller suite still pass (particularly any tests that assert Spec.Replicas == 1, which will need updating in TASK-008 or as part of this task)

Out of Scope

• Reconciler-level validation for VirtualMCPServer session storage (warning status condition when replicas > 1 without Redis) — this is owned by the VirtualMCPServer reconciler and is out of scope for this deployment builder task (see TASK-004 for the MCPServer equivalent; a similar reconciler validation path for VirtualMCPServer may be required in a follow-up)
• Writing formal unit tests — covered by TASK-008
• VirtualMCPServer vMCP ConfigMap Redis injection — covered by TASK-007
• Any changes to mcpserver_controller.go or mcpserver_runconfig.go
• Redis deployment or lifecycle management
• Application-level session storage implementation inside the vMCP process

References

• Epic THV-0047: Horizontal Scaling - CRD and Operator Changes — https://github.com/stacklok/stacklok-epics/issues/264
• Upstream TASK-003: Regenerate deepcopy and CRD manifests — https://github.com/stacklok/stacklok-epics/issues/275
• RFC THV-0047: Manual Horizontal Scaling for vMCP and Proxy Runner — RFC: Horizontal Scaling for vMCP and Proxy Runner toolhive-rfcs#47
• buildOIDCEnvVars reference pattern: cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 289–329
• buildHMACSecretEnvVar reference pattern: cmd/thv-operator/controllers/virtualmcpserver_deployment.go lines 331–348
• Deployment update (replica preservation): cmd/thv-operator/controllers/virtualmcpserver_controller.go lines 926–937