-
Notifications
You must be signed in to change notification settings - Fork 614
[TESTING][CONFIGURATION]: Environment Variables, Validation, and Default Values #2478
Description
[TESTING][CONFIGURATION]: Environment Variables, Validation, and Default Values
Goal
Produce a comprehensive manual test plan for validating configuration handling is robust and user-friendly including environment variable validation, sensible defaults, clear error messages, and secret protection.
Why Now?
Configuration is the first point of failure:
- First Run Experience: Misconfiguration causes frustrating failures
- Error Clarity: Unclear errors waste debugging time
- Security: Secrets must not leak to logs
- Production Safety: Defaults should be safe for production
- Documentation Accuracy: .env.example must match reality
User Stories
US-1: Developer - Clear Error Messages
As a developer
I want clear error messages for invalid configuration
So that I can fix issues quickly
Acceptance Criteria:
Feature: Configuration Validation
Scenario: Missing required variable
Given DATABASE_URL is not set
When I start the gateway
Then I should see "DATABASE_URL is required"
And the process should exit with code 1US-2: Security - Secret Protection
As a security engineer
I want secrets to not appear in logs
So that credentials are protected
Acceptance Criteria:
Feature: Secret Protection
Scenario: JWT secret not logged
Given JWT_SECRET_KEY is set
When the gateway starts and logs configuration
Then JWT_SECRET_KEY value should be masked
And not appear in any log outputArchitecture
CONFIGURATION FLOW
+------------------------------------------------------------------------+
| |
| Environment Validation Application |
| ----------- ---------- ----------- |
| |
| +-----------+ +------------+ +------------+ |
| | .env |------->| Pydantic |------->| Config | |
| | file | | Settings | | Object | |
| +-----------+ +------------+ +------------+ |
| | | | |
| v v v |
| +-----------+ +------------+ +------------+ |
| | Shell | | Type | | Masked | |
| | Env Vars | | Coercion | | Logging | |
| +-----------+ +------------+ +------------+ |
| | | |
| v v |
| +-----------+ +------------+ |
| | Defaults | | Error | |
| | Applied | | Messages | |
| +-----------+ +------------+ |
| |
+------------------------------------------------------------------------+
Test Environment Setup
# Clean environment for testing
unset $(env | grep MCPGATEWAY | cut -d= -f1)
unset DATABASE_URL REDIS_URL JWT_SECRET_KEY
# Ensure .env.example exists
cat .env.example
# Test directory
mkdir -p /tmp/config-test
cd /tmp/config-testManual Test Cases
| Case | Scenario | Input | Expected Result |
|---|---|---|---|
| CFG-01 | Missing required | No DATABASE_URL | Clear error message |
| CFG-02 | Invalid value | PORT=abc | Validation error |
| CFG-03 | Type coercion | RELOAD=true (string) | Works correctly |
| CFG-04 | Defaults applied | Minimal config | Runs with defaults |
| CFG-05 | Secret masking | JWT_SECRET in env | Not logged |
| CFG-06 | Boolean parsing | AUTH_REQUIRED=false | Correctly false |
| CFG-07 | URL validation | DATABASE_URL invalid | Clear error |
| CFG-08 | .env.example sync | All vars | Documented matches |
CFG-01: Missing Required Variable
Steps:
# Unset required variable
unset DATABASE_URL
unset JWT_SECRET_KEY
# Attempt to start
python -m mcpgateway.main 2>&1 | tee /tmp/startup.logExpected Output:
Error: Configuration validation failed:
- DATABASE_URL: Field required
- JWT_SECRET_KEY: Field required
Please set these environment variables or add them to your .env file.
See .env.example for reference.
Validation:
# Check exit code
echo $? # Should be 1
# Check error is clear
grep -q "DATABASE_URL" /tmp/startup.log && echo "PASS: Error mentions variable"
grep -q "required" /tmp/startup.log && echo "PASS: Error says required"Expected Result:
- Process exits with code 1
- Error message names the missing variable
- Error suggests checking .env.example
CFG-02: Invalid Value Type
Steps:
# Set invalid port
export PORT="not-a-number"
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="test-secret"
# Attempt to start
python -m mcpgateway.main 2>&1 | tee /tmp/startup.logExpected Output:
Error: Configuration validation failed:
- PORT: Input should be a valid integer, unable to parse string as an integer
Validation:
grep -q "PORT" /tmp/startup.log && echo "PASS: Error mentions PORT"
grep -q "integer" /tmp/startup.log && echo "PASS: Error mentions expected type"Expected Result:
- Clear error about type mismatch
- Suggests expected type (integer)
CFG-03: Type Coercion
Steps:
# Set boolean as string
export RELOAD="true"
export AUTH_REQUIRED="True"
export MCPGATEWAY_UI_ENABLED="1"
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="test-secret"
# Start gateway
python -m mcpgateway.main &
PID=$!
sleep 5
# Verify running
curl -s http://localhost:8000/health | jq '.status'Validation:
# Verify settings applied correctly
curl -s http://localhost:8000/api/admin/config \
-H "Authorization: Bearer $TOKEN" | jq '{reload, auth_required, ui_enabled}'Expected Result:
- String "true", "True", "1" all convert to boolean true
- Gateway starts successfully
- Settings reflect intended values
CFG-04: Defaults Applied
Steps:
# Minimal configuration
export DATABASE_URL="sqlite:///test.db"
export JWT_SECRET_KEY="minimal-test-secret"
# Start with minimal config
python -m mcpgateway.main &
PID=$!
sleep 5
# Check defaults
curl -s http://localhost:8000/health | jq .Validation:
# Verify defaults
# PORT should default to 8000
curl -s http://localhost:8000/health && echo "PASS: Default port 8000"
# HOST should default to 0.0.0.0 or 127.0.0.1
netstat -tlnp | grep 8000Expected Result:
- Gateway starts with sensible defaults
- Default port is 8000
- Default host is bindable
CFG-05: Secret Masking
Steps:
# Set identifiable secret
export JWT_SECRET_KEY="super-secret-value-12345"
export DATABASE_URL="postgresql://user:secretpassword@localhost/db"
# Start with DEBUG logging
export LOG_LEVEL=DEBUG
python -m mcpgateway.main 2>&1 | tee /tmp/startup.log &
PID=$!
sleep 10
kill $PIDValidation:
# Search for secret in logs
grep -q "super-secret-value-12345" /tmp/startup.log && \
echo "FAIL: Secret in logs!" || \
echo "PASS: Secret not in logs"
grep -q "secretpassword" /tmp/startup.log && \
echo "FAIL: DB password in logs!" || \
echo "PASS: DB password not in logs"
# Check for masking
grep -q "***" /tmp/startup.log && echo "PASS: Masking present"Expected Result:
- Secrets never appear in logs
- Masked with asterisks or "[REDACTED]"
- Database URLs have passwords masked
CFG-06: Boolean Parsing Variants
Steps:
# Test various boolean representations
for val in "true" "True" "TRUE" "1" "yes" "on"; do
export AUTH_REQUIRED="$val"
python -c "from mcpgateway.config import get_settings; print(f'{val} -> {get_settings().auth_required}')"
done
for val in "false" "False" "FALSE" "0" "no" "off" ""; do
export AUTH_REQUIRED="$val"
python -c "from mcpgateway.config import get_settings; print(f'{val} -> {get_settings().auth_required}')" 2>/dev/null || echo "$val -> error"
doneExpected Result:
- All truthy values parse to True
- All falsy values parse to False
- Empty string has defined behavior
CFG-07: URL Validation
Steps:
# Test invalid URLs
export DATABASE_URL="not-a-valid-url"
python -m mcpgateway.main 2>&1 | head -20
export DATABASE_URL="postgresql://missing-host"
python -m mcpgateway.main 2>&1 | head -20
export DATABASE_URL="mysql://unsupported"
python -m mcpgateway.main 2>&1 | head -20Expected Result:
- Invalid URL format rejected with clear error
- Missing components identified
- Unsupported schemes rejected with suggestion
CFG-08: .env.example Synchronization
Steps:
# Extract all env vars from code
grep -r "os.getenv\|os.environ\|settings\." mcpgateway/ | \
grep -oE '[A-Z_]{3,}' | sort -u > /tmp/code_vars.txt
# Extract vars from .env.example
grep -E "^[A-Z]" .env.example | cut -d= -f1 | sort -u > /tmp/example_vars.txt
# Compare
diff /tmp/code_vars.txt /tmp/example_vars.txtValidation:
# Check each .env.example var has description
while read var; do
grep -B1 "^$var=" .env.example | grep -q "#" && \
echo "PASS: $var has description" || \
echo "WARN: $var missing description"
done < /tmp/example_vars.txtExpected Result:
- All used env vars documented in .env.example
- Each variable has a comment describing its purpose
- Default values shown where applicable
Test Matrix
| Variable | Type | Required | Default | Validation |
|---|---|---|---|---|
| DATABASE_URL | URL | Yes | None | Valid DB URL |
| JWT_SECRET_KEY | Secret | Yes | None | Min length |
| HOST | String | No | 0.0.0.0 | Valid IP/hostname |
| PORT | Integer | No | 8000 | 1-65535 |
| RELOAD | Boolean | No | false | truthy/falsy |
| AUTH_REQUIRED | Boolean | No | true | truthy/falsy |
| LOG_LEVEL | Enum | No | INFO | DEBUG/INFO/WARNING/ERROR |
| REDIS_URL | URL | No | None | Valid Redis URL |
Success Criteria
- All required env vars documented in .env.example
- Missing required vars fail with clear error message
- Invalid values fail with type-specific error
- Type coercion works for booleans and integers
- Defaults are production-safe
- Secrets never appear in logs (masked)
- .env.example is synchronized with code
- Config validation runs at startup (fail fast)
Related Files
mcpgateway/config.py- Configuration settings.env.example- Environment templatemcpgateway/main.py- Startup validationmcpgateway/utils/logging.py- Log masking
Related Issues
- [TESTING][FUNCTIONALITY]: Configuration manual test plan (env vars, runtime config, validation) #2434 - Configuration (functionality)
- [TESTING][DEPLOYMENT]: Docker, Docker Compose, Kubernetes/Helm, and Bare Metal Installation #2475 - Deployment testing
- [TESTING][SECURITY]: Encryption and secrets manual test plan (Argon2, Fernet, key derivation) #2405 - Encryption and secrets