Skip to content

[TESTING][CONFIGURATION]: Environment Variables, Validation, and Default Values #2478

@crivetimihai

Description

@crivetimihai

[TESTING][CONFIGURATION]: Environment Variables, Validation, and Default Values

Goal

Produce a comprehensive manual test plan for validating configuration handling is robust and user-friendly including environment variable validation, sensible defaults, clear error messages, and secret protection.

Why Now?

Configuration is the first point of failure:

  1. First Run Experience: Misconfiguration causes frustrating failures
  2. Error Clarity: Unclear errors waste debugging time
  3. Security: Secrets must not leak to logs
  4. Production Safety: Defaults should be safe for production
  5. Documentation Accuracy: .env.example must match reality

User Stories

US-1: Developer - Clear Error Messages

As a developer
I want clear error messages for invalid configuration
So that I can fix issues quickly

Acceptance Criteria:

Feature: Configuration Validation

  Scenario: Missing required variable
    Given DATABASE_URL is not set
    When I start the gateway
    Then I should see "DATABASE_URL is required"
    And the process should exit with code 1
US-2: Security - Secret Protection

As a security engineer
I want secrets to not appear in logs
So that credentials are protected

Acceptance Criteria:

Feature: Secret Protection

  Scenario: JWT secret not logged
    Given JWT_SECRET_KEY is set
    When the gateway starts and logs configuration
    Then JWT_SECRET_KEY value should be masked
    And not appear in any log output

Architecture

                    CONFIGURATION FLOW
+------------------------------------------------------------------------+
|                                                                        |
|   Environment           Validation           Application               |
|   -----------           ----------           -----------               |
|                                                                        |
|   +-----------+        +------------+        +------------+            |
|   |  .env     |------->|  Pydantic  |------->|  Config    |            |
|   |  file     |        |  Settings  |        |  Object    |            |
|   +-----------+        +------------+        +------------+            |
|        |                     |                     |                   |
|        v                     v                     v                   |
|   +-----------+        +------------+        +------------+            |
|   |  Shell    |        |  Type      |        |  Masked    |            |
|   |  Env Vars |        |  Coercion  |        |  Logging   |            |
|   +-----------+        +------------+        +------------+            |
|        |                     |                                         |
|        v                     v                                         |
|   +-----------+        +------------+                                  |
|   |  Defaults |        |  Error     |                                  |
|   |  Applied  |        |  Messages  |                                  |
|   +-----------+        +------------+                                  |
|                                                                        |
+------------------------------------------------------------------------+

Test Environment Setup

# Clean environment for testing
unset $(env | grep MCPGATEWAY | cut -d= -f1)
unset DATABASE_URL REDIS_URL JWT_SECRET_KEY

# Ensure .env.example exists
cat .env.example

# Test directory
mkdir -p /tmp/config-test
cd /tmp/config-test

Manual Test Cases

Case Scenario Input Expected Result
CFG-01 Missing required No DATABASE_URL Clear error message
CFG-02 Invalid value PORT=abc Validation error
CFG-03 Type coercion RELOAD=true (string) Works correctly
CFG-04 Defaults applied Minimal config Runs with defaults
CFG-05 Secret masking JWT_SECRET in env Not logged
CFG-06 Boolean parsing AUTH_REQUIRED=false Correctly false
CFG-07 URL validation DATABASE_URL invalid Clear error
CFG-08 .env.example sync All vars Documented matches

CFG-01: Missing Required Variable

Steps:

# Unset required variable
unset DATABASE_URL
unset JWT_SECRET_KEY

# Attempt to start
python -m mcpgateway.main 2>&1 | tee /tmp/startup.log

Expected Output:

Error: Configuration validation failed:
  - DATABASE_URL: Field required
  - JWT_SECRET_KEY: Field required

Please set these environment variables or add them to your .env file.
See .env.example for reference.

Validation:

# Check exit code
echo $?  # Should be 1

# Check error is clear
grep -q "DATABASE_URL" /tmp/startup.log && echo "PASS: Error mentions variable"
grep -q "required" /tmp/startup.log && echo "PASS: Error says required"

Expected Result:

  • Process exits with code 1
  • Error message names the missing variable
  • Error suggests checking .env.example
CFG-02: Invalid Value Type

Steps:

# Set invalid port
export PORT="not-a-number"
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="test-secret"

# Attempt to start
python -m mcpgateway.main 2>&1 | tee /tmp/startup.log

Expected Output:

Error: Configuration validation failed:
  - PORT: Input should be a valid integer, unable to parse string as an integer

Validation:

grep -q "PORT" /tmp/startup.log && echo "PASS: Error mentions PORT"
grep -q "integer" /tmp/startup.log && echo "PASS: Error mentions expected type"

Expected Result:

  • Clear error about type mismatch
  • Suggests expected type (integer)
CFG-03: Type Coercion

Steps:

# Set boolean as string
export RELOAD="true"
export AUTH_REQUIRED="True"
export MCPGATEWAY_UI_ENABLED="1"
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="test-secret"

# Start gateway
python -m mcpgateway.main &
PID=$!
sleep 5

# Verify running
curl -s http://localhost:8000/health | jq '.status'

Validation:

# Verify settings applied correctly
curl -s http://localhost:8000/api/admin/config \
  -H "Authorization: Bearer $TOKEN" | jq '{reload, auth_required, ui_enabled}'

Expected Result:

  • String "true", "True", "1" all convert to boolean true
  • Gateway starts successfully
  • Settings reflect intended values
CFG-04: Defaults Applied

Steps:

# Minimal configuration
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="minimal-test-secret"

# Start with minimal config
python -m mcpgateway.main &
PID=$!
sleep 5

# Check defaults
curl -s http://localhost:8000/health | jq .

Validation:

# Verify defaults
# PORT should default to 8000
curl -s http://localhost:8000/health && echo "PASS: Default port 8000"

# HOST should default to 0.0.0.0 or 127.0.0.1
netstat -tlnp | grep 8000

Expected Result:

  • Gateway starts with sensible defaults
  • Default port is 8000
  • Default host is bindable
CFG-05: Secret Masking

Steps:

# Set identifiable secret
export JWT_SECRET_KEY="super-secret-value-12345"
export DATABASE_URL="postgresql://user:secretpassword@localhost/db"

# Start with DEBUG logging
export LOG_LEVEL=DEBUG
python -m mcpgateway.main 2>&1 | tee /tmp/startup.log &
PID=$!
sleep 10
kill $PID

Validation:

# Search for secret in logs
grep -q "super-secret-value-12345" /tmp/startup.log && \
  echo "FAIL: Secret in logs!" || \
  echo "PASS: Secret not in logs"

grep -q "secretpassword" /tmp/startup.log && \
  echo "FAIL: DB password in logs!" || \
  echo "PASS: DB password not in logs"

# Check for masking
grep -q "***" /tmp/startup.log && echo "PASS: Masking present"

Expected Result:

  • Secrets never appear in logs
  • Masked with asterisks or "[REDACTED]"
  • Database URLs have passwords masked
CFG-06: Boolean Parsing Variants

Steps:

# Test various boolean representations
for val in "true" "True" "TRUE" "1" "yes" "on"; do
  export AUTH_REQUIRED="$val"
  python -c "from mcpgateway.config import get_settings; print(f'{val} -> {get_settings().auth_required}')"
done

for val in "false" "False" "FALSE" "0" "no" "off" ""; do
  export AUTH_REQUIRED="$val"
  python -c "from mcpgateway.config import get_settings; print(f'{val} -> {get_settings().auth_required}')" 2>/dev/null || echo "$val -> error"
done

Expected Result:

  • All truthy values parse to True
  • All falsy values parse to False
  • Empty string has defined behavior
CFG-07: URL Validation

Steps:

# Test invalid URLs
export DATABASE_URL="not-a-valid-url"
python -m mcpgateway.main 2>&1 | head -20

export DATABASE_URL="postgresql://missing-host"
python -m mcpgateway.main 2>&1 | head -20

export DATABASE_URL="mysql://unsupported"
python -m mcpgateway.main 2>&1 | head -20

Expected Result:

  • Invalid URL format rejected with clear error
  • Missing components identified
  • Unsupported schemes rejected with suggestion
CFG-08: .env.example Synchronization

Steps:

# Extract all env vars from code
grep -r "os.getenv\|os.environ\|settings\." mcpgateway/ | \
  grep -oE '[A-Z_]{3,}' | sort -u > /tmp/code_vars.txt

# Extract vars from .env.example
grep -E "^[A-Z]" .env.example | cut -d= -f1 | sort -u > /tmp/example_vars.txt

# Compare
diff /tmp/code_vars.txt /tmp/example_vars.txt

Validation:

# Check each .env.example var has description
while read var; do
  grep -B1 "^$var=" .env.example | grep -q "#" && \
    echo "PASS: $var has description" || \
    echo "WARN: $var missing description"
done < /tmp/example_vars.txt

Expected Result:

  • All used env vars documented in .env.example
  • Each variable has a comment describing its purpose
  • Default values shown where applicable

Test Matrix

Variable Type Required Default Validation
DATABASE_URL URL Yes None Valid DB URL
JWT_SECRET_KEY Secret Yes None Min length
HOST String No 0.0.0.0 Valid IP/hostname
PORT Integer No 8000 1-65535
RELOAD Boolean No false truthy/falsy
AUTH_REQUIRED Boolean No true truthy/falsy
LOG_LEVEL Enum No INFO DEBUG/INFO/WARNING/ERROR
REDIS_URL URL No None Valid Redis URL

Success Criteria

  • All required env vars documented in .env.example
  • Missing required vars fail with clear error message
  • Invalid values fail with type-specific error
  • Type coercion works for booleans and integers
  • Defaults are production-safe
  • Secrets never appear in logs (masked)
  • .env.example is synchronized with code
  • Config validation runs at startup (fail fast)

Related Files

  • mcpgateway/config.py - Configuration settings
  • .env.example - Environment template
  • mcpgateway/main.py - Startup validation
  • mcpgateway/utils/logging.py - Log masking

Related Issues

Metadata

Metadata

Labels

MUSTP1: Non-negotiable, critical requirements without which the product is non-functional or unsafechoreLinting, formatting, dependency hygiene, or project maintenance choresmanual-testingManual testing / test planning issuesreadyValidated, ready-to-work-on itemstestingTesting (unit, e2e, manual, automated, etc)

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions