note: working atop original #1059 work to handle CI pipeline fixes

_currently working on resolving linting issues + test failures, then will go over copilot (which a lot is relevant to linting fails)
todo: after above, will re-validate it works as expected using attached doc (so testing tls + existing agents (which would ensure backwards compat.)

tls-validation.md

major changes from original pr

• my feedback review
• test setup changes + fixes
• lint resolutions
• update to check on when we create ssl context to include when disabling system cas
• untracking test certs, dynamically generating for tests (in ci) + makefile target for local
  • just pushed, need to make sure it runs well in ci

(original description)

Summary

This PR adds comprehensive SSL/TLS configuration support to Kagent's ModelConfig CRD, enabling agents to securely connect to internal LiteLLM gateways and model providers that use self-signed certificates or custom certificate authorities.

Note: TLS configuration is currently only implemented for OpenAI-compatible model types (OpenAI and AzureOpenAI providers). This design specifically targets internal LiteLLM gateway deployments. The field structure is intentionally generic to facilitate future implementations for other model types that require custom certificate handling.

This is a production-ready, Kubernetes-native implementation that follows security best practices and maintains full backward compatibility with existing ModelConfig resources.

Problem Statement

Organizations running Kagent often need to connect agents to:

• Internal LiteLLM gateways with self-signed certificates
• Model providers behind corporate proxies with custom CAs
• Development/staging environments with non-production certificates

Previously, there was no way to configure custom CA certificates or disable SSL verification for these scenarios, forcing users to:

• Modify container images to trust custom CAs (non-scalable)
• Use insecure workarounds that bypass SSL entirely (security risk)
• Deploy public certificates for internal services (operational overhead)

Solution

This PR introduces a new tls field in the ModelConfig spec that supports three modes:

1. Disabled Verification (Development/Testing Only)

spec:
  provider: OpenAI  # Required: TLS only works with OpenAI/AzureOpenAI
  tls:
    disableVerify: true

Disables SSL verification entirely. Includes security warnings in logs.

2. Custom CA Only

spec:
  provider: OpenAI  # Required: TLS only works with OpenAI/AzureOpenAI
  tls:
    caCertSecretRef: litellm-ca-cert
    caCertSecretKey: ca.crt
    disableSystemCAs: true

Trust only the specified CA certificate from a Kubernetes Secret.

3. System + Custom CA (Recommended)

spec:
  provider: OpenAI  # Required: TLS only works with OpenAI/AzureOpenAI
  tls:
    caCertSecretRef: litellm-ca-cert
    caCertSecretKey: ca.crt
    disableSystemCAs: false  # default - trust both system and custom CAs

Trust both system CAs (for public services) and custom CAs (for internal services). This is the recommended approach for hybrid environments.

Changes Made

Go Backend (Kubernetes CRD & Controller)

CRD Schema (v1alpha2 only)

• Removed TLS from v1alpha1 - TLS configuration only exists in v1alpha2

• Added TLSConfig struct with four fields:

  • disableVerify (bool): Disable SSL verification (default: false)
  • caCertSecretRef (string): Reference to Secret containing CA cert
  • caCertSecretKey (string): Key within Secret (default: "ca.crt")
  • disableSystemCAs (bool): When true, only trust custom CAs (default: false)

• Added CEL validation rules for field consistency

• Updated CRD manifests with OpenAPI schema

• Generated deepcopy methods

• Note: All field names follow the "falsey-by-default" pattern where false = safe/secure behavior

Files changed:

• go/api/v1alpha2/modelconfig_types.go
• go/config/crd/bases/kagent.dev_modelconfigs.yaml

Kubernetes Controller

• Changed from environment variables to agent config JSON - TLS configuration is now passed through /config/config.json instead of environment variables

• Implemented addTLSConfiguration() function to mount TLS certificates

• Controller automatically:

  • Mounts CA certificate Secrets as volumes at /etc/ssl/certs/custom/
  • Passes TLS config through agent config JSON with fields: tls_disable_verify, tls_ca_cert_path, tls_disable_system_cas
  • Creates read-only volume mounts with mode 0444
  • Handles missing or incomplete TLS config gracefully (no-op when nil)

Files changed:

• go/internal/controller/translator/agent/adk_api_translator.go
• go/internal/adk/types.go

Test Coverage (7 test functions)

• Controller mounting tests: 7 test scenarios covering volume mounts, config propagation, error cases

Test files:

• go/internal/controller/translator/agent/tls_mounting_test.go

Python Runtime (kagent-adk)

SSL Utilities Module

• Created _ssl.py with create_ssl_context() function

• Supports three TLS modes:

  1. Disabled verification (returns False, logs security warnings)
  2. Custom CA only (loads CA cert, creates SSLContext)
  3. System + Custom CA (uses default certifi certs + custom CA)

• Certificate validation with clear error messages

• Structured logging for audit trail and troubleshooting

File:

• python/packages/kagent-adk/src/kagent/adk/models/_ssl.py

OpenAI SDK Integration (OpenAI/AzureOpenAI Only)

• Extended BaseOpenAI and AzureOpenAI classes with TLS fields:

  • tls_disable_verify, tls_ca_cert_path, tls_disable_system_cas

• Added _get_tls_config() to read from agent config

• Added _create_http_client() to build custom httpx.AsyncClient with SSL context

• AsyncOpenAI and AsyncAzureOpenAI use custom http_client when TLS configured

• Falls back to SDK defaults when no TLS configuration present (backward compatible)

• Note: TLS is only implemented for OpenAI and AzureOpenAI model types

Files changed:

• python/packages/kagent-adk/src/kagent/adk/models/_openai.py

Type System

• Added TLS fields to BaseLLM (available to all model types for future extensibility)
• TLS fields used in OpenAI and AzureOpenAI Pydantic models
• Extended AgentConfig.to_agent() to propagate TLS config to model instances
• Type-safe configuration with optional fields (fully backward compatible)

Files changed:

• python/packages/kagent-adk/src/kagent/adk/types.py

Test Coverage (26 tests passing)

• test_ssl.py: SSL context creation, certificate loading, error handling
• test_openai.py: OpenAI client instantiation with TLS
• test_tls_integration.py: End-to-end OpenAI/Azure integration
• test_tls_e2e.py: Full workflow with mock HTTPS servers
• Test fixtures: Self-signed CA and server certificates for realistic testing

Test files:

• python/packages/kagent-adk/tests/unittests/models/test_ssl.py
• python/packages/kagent-adk/tests/unittests/models/test_openai.py
• python/packages/kagent-adk/tests/unittests/models/test_tls_integration.py
• python/packages/kagent-adk/tests/unittests/models/test_tls_e2e.py
• python/packages/kagent-adk/tests/fixtures/certs/

Examples

YAML Examples (examples/modelconfig-with-tls.yaml):

• Complete working examples for all three modes
• Secret creation examples
• Commented YAML with explanations
• All examples include provider: OpenAI requirement

Key Features

1. Kubernetes-Native Design

• Uses Kubernetes Secrets for certificate storage (follows best practices)
• Volume mounts for certificate access (secure, standard pattern)
• Configuration passed through agent config JSON (not environment variables)
• CEL validation at admission time

2. Security-Focused

• Secrets stored encrypted at rest by Kubernetes
• Read-only volume mounts (mode 0444)
• Certificate validation with clear error messages
• Security warnings for disabled verification in logs
• Falsey-by-default field naming for safe defaults

3. Production-Ready

• Comprehensive error handling and validation
• Structured logging for audit trail and debugging
• Fully backward compatible (existing configs unchanged)
• Extensive test coverage (33 test functions)
• OpenAI-only implementation limits scope and complexity

4. Developer-Friendly

• Clear examples in YAML and Python
• Environment variable overrides for local development
• Extensible field structure for future model type implementations

Provider Support

Currently Supported:

• ✅ OpenAI (native)
• ✅ AzureOpenAI
• ✅ LiteLLM (via OpenAI-compatible API)

Not Yet Supported:

• ❌ Anthropic
• ❌ Google Gemini
• ❌ Ollama
• ❌ Other providers

The TLS configuration fields are defined in BaseLLM to facilitate future implementations, but only OpenAI and AzureOpenAI model types currently use them. If custom certificate handling is needed for other providers, implementations can reuse the same field structure.

Testing

All tests pass:

• Go tests: 7 TLS-specific test functions
• Python tests: 26 tests passing, 4 skipped (expected)

Run tests:

# Go tests
cd go && go test ./internal/controller/translator/agent -run TestTLS -v

# Python tests  
cd python/packages/kagent-adk
pytest tests/unittests/models/test_ssl.py -v
pytest tests/unittests/models/test_openai.py -v
pytest tests/unittests/models/test_tls_integration.py -v
pytest tests/unittests/models/test_tls_e2e.py -v

Usage Example

1. Create a Secret with your CA certificate:

kubectl create secret generic litellm-ca-cert \
  --from-file=ca.crt=/path/to/your/ca.crt \
  -n kagent

2. Create a ModelConfig with TLS configuration:

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: litellm-with-custom-ca
  namespace: kagent
spec:
  provider: OpenAI  # Required: TLS only works with OpenAI/AzureOpenAI
  model: gpt-4
  apiKeySecretRef: openai-api-key
  apiKeySecretKey: key
  openAI:
    baseUrl: https://litellm.internal.company.com
  tls:
    caCertSecretRef: litellm-ca-cert
    caCertSecretKey: ca.crt
    disableSystemCAs: false  # Trust both system CAs and custom CA

3. Use the ModelConfig in your Agent:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: my-agent
spec:
  framework: ADK
  modelConfigName: litellm-with-custom-ca
  card:
    name: my-agent
    description: Agent using internal LiteLLM gateway

The agent will now be able to connect to the internal LiteLLM gateway using the custom CA certificate!

Breaking Changes

None. This is a purely additive feature.

• Existing ModelConfig resources without tls field continue to work unchanged
• Default behavior is unchanged (standard SSL verification)
• No migration required for existing deployments
• Backward compatible API changes (optional fields only)
• TLS only exists in v1alpha2 (v1alpha1 unchanged)

Migration

No migration required. The tls field is optional with safe defaults:

• disableVerify defaults to false (verification enabled - secure)
• disableSystemCAs defaults to false (trust system CAs - safe)
• Agents without tls configuration use standard SSL verification
• Existing ModelConfigs work exactly as before

Security Considerations

Best Practices

1. Never disable SSL verification in production - Use disableVerify: true only for development/testing
2. Use Kubernetes Secrets for CA certificates - Never embed certificates in ConfigMaps or code
3. Set up proper RBAC - Limit Secret access to authorized ServiceAccounts only
4. Rotate certificates regularly - Update Secrets when certificates expire
5. Monitor logs - Watch for SSL warnings and certificate expiration notices
6. Use disableSystemCAs: false - Recommended (default) to maintain trust in public CAs

Security Features

• Certificate validation with clear error messages
• Security warnings logged when verification is disabled
• Read-only volume mounts (no write access to certificates)
• Secrets encrypted at rest by Kubernetes
• Falsey-by-default naming: false = secure behavior

Field Naming Rationale

All boolean fields follow the falsey-by-default pattern:

• disableVerify: false = verification enabled (secure) ✅
• disableSystemCAs: false = system CAs enabled (safe) ✅

This ensures that omitting fields or using default values results in the most secure configuration.

Review Checklist

• ✅ Kubernetes CRD changes: TLSConfig struct added to v1alpha2 only
• ✅ Controller logic: Volume mounting and agent config JSON propagation
• ✅ Python runtime: SSL context creation and OpenAI client integration (OpenAI/AzureOpenAI only)
• ✅ Type safety: Pydantic models with optional TLS fields
• ✅ Validation: CEL validation rules for field consistency
• ✅ Error handling: Clear error messages for certificate and configuration issues
• ✅ Logging: Structured logging with security warnings
• ✅ Test coverage: 33 test functions covering all scenarios
• ✅ Backward compatibility: No breaking changes, existing configs work unchanged
• ✅ Security: Secrets, validation, warnings, falsey-by-default naming
• ✅ Provider scope: OpenAI/AzureOpenAI only, documented clearly

Next Steps

After this PR is merged:

1. Deploy updated CRDs to cluster (kubectl apply -f go/config/crd/bases/)
2. Update Kagent controller deployment with new image
3. Update kagent-adk package in agent images
4. Share documentation with teams needing TLS configuration
5. Monitor logs for SSL warnings in development environments