🚀 Epic: Performance Testing & Benchmarking Framework

Goal

Enable comprehensive performance testing, benchmarking, and regression detection for MCP Gateway, providing automated tooling to measure throughput, latency, resource utilization, and scalability across all MCP protocol operations. This ensures production readiness, identifies performance bottlenecks, and prevents regressions in critical API paths.

Why Now?

Performance is critical for production adoption of MCP Gateway. As the gateway federates multiple MCP servers, handles high-volume tool invocations, and serves as a central API hub, we need:

1. Quantifiable performance baselines to understand current system capabilities
2. Regression detection to catch performance degradations before production
3. Scalability insights to guide infrastructure planning and optimization
4. Database vs. in-memory trade-offs to optimize different deployment profiles
5. Load testing automation to validate production readiness

Without systematic performance testing, we risk shipping performance regressions, failing to meet production SLAs, and making uninformed architectural decisions.

This framework positions MCP Gateway as a production-grade, performance-verified platform.

📖 User Stories

Given MCP Gateway is running locally or in Docker
When I execute "make test" from tests/performance/
Then the system should:
  - Verify all required services are running (gateway, database, cache)
  - Authenticate and obtain JWT tokens
  - Run benchmarks against all core endpoints (/tools, /resources, /prompts, /servers)
  - Generate detailed HTML reports with response time distributions
  - Compare results against baseline metrics
  - Flag performance regressions >10% degradation
  - Save new baseline if performance improved
  - Exit with code 0 if tests pass, non-zero if regressions detected

Given performance profiles defined (light.env, medium.env, heavy.env)
When I execute "./run-configurable.sh --profile heavy --scenarios all"
Then the system should:
  - Load heavy profile: 5000 requests, 200 concurrent users
  - Run all scenarios: tools, resources, prompts, mixed-workload, database, gateway-core
  - Test with 4 simulated MCP servers (SQLite + PostgreSQL)
  - Generate comparative HTML report
  - Show P50/P95/P99 latencies for each scenario
  - Display requests/sec and total duration
  - Compare against baseline for heavy profile
  - Store results in results/ directory with timestamp

Given database benchmarking scenario
When I execute "./scenarios/database-benchmark.sh"
Then the system should:
  - Test direct database query performance (list tools, servers, prompts)
  - Test database write operations (create/update/delete)
  - Test complex joins (tool + team + server queries)
  - Compare SQLite vs PostgreSQL performance
  - Generate database-specific performance report
  - Identify slow queries (>100ms)
  - Provide optimization recommendations

Given baseline metrics stored from previous release
When CI runs performance tests on new commit
Then the system should:
  - Load baseline metrics from baselines/baseline_<profile>.json
  - Run full test suite against new code
  - Compare each metric against baseline
  - Calculate percentage change (positive = improvement, negative = regression)
  - Flag any metric with >10% regression
  - Generate GitHub Actions annotation for regressions
  - Block merge if critical regressions detected
  - Update baseline if improvements >5% sustained

Given scalability test configuration
When I execute "./run-advanced.sh --servers 10 --profile heavy"
Then the system should:
  - Generate docker-compose.yaml with 10 MCP servers
  - Start all servers (5 SQLite, 5 PostgreSQL)
  - Register all servers with gateway
  - Run mixed workload across all servers
  - Measure aggregated throughput (RPS)
  - Measure resource utilization (CPU, memory, DB connections)
  - Test horizontal scaling limits
  - Generate scalability report with server count vs RPS correlation

Given MANUAL_TESTING.md guide
When I run CLI commands from the guide
Then I should be able to:
  - Login and obtain JWT token via curl
  - List tools/resources/prompts with jq filtering
  - Run quick performance tests with hey (100-1000 requests)
  - Monitor API health continuously
  - Test authentication token expiration
  - Benchmark specific endpoints individually
  - Verify deployment after code changes

🏗 Architecture

Testing Framework Architecture

graph TB
    subgraph "Test Orchestration"
        A1[Makefile]
        A2[run-configurable.sh]
        A3[run-advanced.sh]
        A4[run-all.sh]
    end

    subgraph "Configuration"
        B1[config.yaml]
        B2[profiles/light.env]
        B3[profiles/medium.env]
        B4[profiles/heavy.env]
    end

    subgraph "Scenarios"
        C1[tools-benchmark.sh]
        C2[resources-benchmark.sh]
        C3[prompts-benchmark.sh]
        C4[mixed-workload.sh]
        C5[database-benchmark.sh]
        C6[gateway-core-benchmark.sh]
    end

    subgraph "Utilities"
        D1[setup-auth.sh]
        D2[check-services.sh]
        D3[baseline_manager.py]
        D4[compare_results.py]
        D5[report_generator.py]
        D6[generate_docker_compose.py]
    end

    subgraph "MCP Gateway"
        E1[REST API]
        E2[Database]
        E3[MCP Servers]
    end

    A1 --> C1
    A2 --> B1
    A2 --> B2
    A3 --> D6
    C1 --> D1
    C1 --> D2
    C1 --> E1
    D3 --> D4
    D4 --> D5
    E1 --> E2
    E1 --> E3

Test Execution Flow

sequenceDiagram
    participant User
    participant Makefile
    participant Setup as setup-auth.sh
    participant Check as check-services.sh
    participant Scenario as Benchmark Script
    participant API as MCP Gateway
    participant Baseline as baseline_manager.py
    participant Compare as compare_results.py
    participant Report as report_generator.py

    User->>Makefile: make test
    Makefile->>Check: Verify services
    Check->>API: GET /health
    API-->>Check: 200 OK
    Check-->>Makefile: Services ready

    Makefile->>Setup: Authenticate
    Setup->>API: POST /auth/login
    API-->>Setup: JWT token
    Setup-->>Makefile: TOKEN exported

    Makefile->>Scenario: Run benchmark
    Scenario->>API: hey -n 1000 -c 50 GET /tools
    API-->>Scenario: Performance metrics
    Scenario->>Scenario: Parse hey output
    Scenario-->>Makefile: results/tools_YYYY-MM-DD-HHMMSS.txt

    Makefile->>Baseline: Load baseline
    Baseline-->>Makefile: baseline_light.json

    Makefile->>Compare: Compare results
    Compare->>Compare: Calculate delta %
    Compare-->>Makefile: comparison_report.txt

    Makefile->>Report: Generate HTML
    Report->>Report: Create charts
    Report-->>Makefile: report.html

    Makefile-->>User: Open report in browser

Baseline Management System

graph LR
    A[Run Tests] --> B{Baseline Exists?}
    B -->|No| C[Save as Initial Baseline]
    B -->|Yes| D[Load Baseline]
    D --> E[Compare Metrics]
    E --> F{Regression?}
    F -->|Yes >10%| G[Fail Tests]
    F -->|No| H{Improvement?}
    H -->|Yes >5%| I[Update Baseline]
    H -->|No| J[Keep Baseline]
    I --> K[Pass Tests]
    J --> K
    C --> K

📂 Directory Structure

tests/performance/
├── Makefile                       # Main test automation (271 lines, 40+ targets)
├── config.yaml                    # Centralized configuration (386 lines)
├── README.md                      # Comprehensive guide (309 lines)
├── MANUAL_TESTING.md              # CLI testing guide (458 lines)
├── PERFORMANCE_STRATEGY.md        # Testing strategy doc (2116 lines)
│
├── profiles/                      # Load profiles
│   ├── light.env                  # 100 req, 10 concurrent
│   ├── medium.env                 # 1000 req, 50 concurrent
│   └── heavy.env                  # 5000 req, 200 concurrent
│
├── scenarios/                     # Benchmark scenarios
│   ├── tools-benchmark.sh         # Test /tools endpoint (134 lines)
│   ├── resources-benchmark.sh     # Test /resources endpoint (134 lines)
│   ├── prompts-benchmark.sh       # Test /prompts endpoint (126 lines)
│   ├── mixed-workload.sh          # Combined workload (152 lines)
│   ├── database-benchmark.sh      # Direct DB tests (217 lines)
│   └── gateway-core-benchmark.sh  # Core operations (268 lines)
│
├── payloads/                      # Test request payloads
│   ├── tools/
│   │   ├── list_tools.json
│   │   ├── get_system_time.json
│   │   └── convert_time.json
│   ├── resources/
│   │   ├── list_resources.json
│   │   ├── read_timezone_info.json
│   │   └── read_world_times.json
│   └── prompts/
│       ├── list_prompts.json
│       └── get_compare_timezones.json
│
├── utils/                         # Python utilities
│   ├── setup-auth.sh              # Authentication helper (88 lines)
│   ├── check-services.sh          # Service health check (67 lines)
│   ├── baseline_manager.py        # Baseline CRUD (323 lines)
│   ├── compare_results.py         # Regression detection (376 lines)
│   ├── report_generator.py        # HTML reports (1195 lines)
│   └── generate_docker_compose.py # Multi-server setup (423 lines)
│
├── baselines/                     # Stored baselines (gitignored)
│   ├── baseline_light.json
│   ├── baseline_medium.json
│   └── baseline_heavy.json
│
├── results/                       # Test results (gitignored)
│   ├── tools_2025-10-10-143000.txt
│   ├── resources_2025-10-10-143100.txt
│   └── report_2025-10-10-143200.html
│
├── run-configurable.sh            # Configurable test runner (435 lines)
├── run-advanced.sh                # Multi-server testing (381 lines)
└── run-all.sh                     # All scenarios (271 lines)

⚙️ Configuration

config.yaml Structure

# Service Configuration
service:
  base_url: "http://localhost:4444"
  health_check_timeout: 30
  auth_required: true

# Authentication
auth:
  email: "admin@example.com"
  password: "changeme"
  token_cache: "/tmp/perf_test_token.txt"

# Performance Settings
performance:
  hey_binary: "hey"
  default_requests: 1000
  default_concurrency: 50
  default_timeout: 30

# Baseline Configuration
baselines:
  storage_dir: "baselines"
  regression_threshold: 10.0  # % degradation before failure
  improvement_threshold: 5.0  # % improvement to update baseline
  metrics:
    - p50_latency
    - p95_latency
    - p99_latency
    - requests_per_sec
    - total_time

# Test Scenarios
scenarios:
  tools:
    endpoint: "/tools"
    methods: [GET, POST]
    payloads_dir: "payloads/tools"
  resources:
    endpoint: "/resources"
    methods: [GET, POST]
    payloads_dir: "payloads/resources"
  # ... more scenarios

# Reporting
reporting:
  output_dir: "results"
  html_template: "utils/report_template.html"
  open_browser: true
  include_charts: true
  chart_types: ["latency_distribution", "throughput_timeline"]

Profile Configuration Example

# profiles/heavy.env
# Heavy load profile for stress testing

# Load Parameters
export PERF_REQUESTS=5000
export PERF_CONCURRENCY=200
export PERF_TIMEOUT=60

# Test Configuration
export PERF_SCENARIOS="tools,resources,prompts,mixed,database,gateway-core"
export PERF_ITERATIONS=3
export PERF_WARMUP=true

# Database Configuration
export PERF_DB_BACKEND="postgresql"
export PERF_SERVERS_COUNT=10

# Reporting
export PERF_BASELINE_PROFILE="heavy"
export PERF_REGRESSION_THRESHOLD=10
export PERF_GENERATE_REPORT=true

📋 Implementation Tasks

Phase 1: Core Infrastructure ✅

• Project Structure

  • Create tests/performance/ directory
  • Add .gitignore for results, baselines, Docker artifacts
  • Create subdirectories: scenarios/, utils/, payloads/, profiles/, baselines/
  • Initialize README.md with overview

• Configuration System

  • Create config.yaml with service, auth, performance sections
  • Define baseline configuration
  • Create profile templates (light.env, medium.env, heavy.env)
  • Add environment variable validation

• Makefile Automation

  • Create Makefile with 40+ targets
  • Add test, test-tools, test-resources, test-prompts targets
  • Implement test-all target for full suite
  • Add clean, clean-results, clean-baselines targets
  • Create help target with command documentation
  • Add Docker targets (test-docker, docker-up, docker-down)

Phase 2: Benchmark Scenarios ✅

• Tools Endpoint Benchmarks

  • Create scenarios/tools-benchmark.sh (134 lines)
  • Test GET /tools with various limits (10, 50, 100)
  • Test POST /tools/invoke with tool execution
  • Parse hey output for metrics
  • Save results with timestamps

• Resources Endpoint Benchmarks

  • Create scenarios/resources-benchmark.sh (134 lines)
  • Test GET /resources listing
  • Test GET /resources/{uri} retrieval
  • Test StreamableHTTP resource types
  • Handle large resource payloads

• Prompts Endpoint Benchmarks

  • Create scenarios/prompts-benchmark.sh (126 lines)
  • Test GET /prompts listing
  • Test GET /prompts/{name} retrieval
  • Test prompt template expansion
  • Measure argument validation overhead

• Mixed Workload Scenario

  • Create scenarios/mixed-workload.sh (152 lines)
  • Run tools + resources + prompts concurrently
  • Simulate realistic API usage patterns
  • Measure aggregate system throughput
  • Test connection pool saturation

• Database Benchmark Scenario

  • Create scenarios/database-benchmark.sh (217 lines)
  • Test direct SQLAlchemy query performance
  • Compare SQLite vs PostgreSQL
  • Benchmark complex joins (tool + team + server)
  • Test connection pool efficiency

• Gateway Core Benchmark

  • Create scenarios/gateway-core-benchmark.sh (268 lines)
  • Test /health endpoint (no auth overhead)
  • Test /servers listing and SSE connections
  • Test gateway federation features
  • Measure middleware overhead

Phase 3: Utilities & Automation ✅

• Authentication Helper

  • Create utils/setup-auth.sh (88 lines)
  • Implement login via POST /auth/login
  • Extract JWT token from JSON response
  • Cache token to /tmp/perf_test_token.txt
  • Validate token expiration
  • Auto-refresh expired tokens

• Service Health Check

  • Create utils/check-services.sh (67 lines)
  • Test gateway availability (GET /health)
  • Verify database connectivity
  • Check Redis/cache availability
  • Validate MCP servers are registered
  • Exit with error code if services unavailable

• Baseline Manager

  • Create utils/baseline_manager.py (323 lines)
  • Implement save_baseline() for storing metrics
  • Implement load_baseline() for retrieval
  • Add list_baselines() for discovery
  • Implement delete_baseline() for cleanup
  • Support profile-specific baselines
  • JSON serialization with pretty printing

• Results Comparison

  • Create utils/compare_results.py (376 lines)
  • Parse hey output format
  • Extract P50, P95, P99 latencies
  • Calculate requests/sec and total time
  • Compare against baseline metrics
  • Calculate percentage delta
  • Flag regressions exceeding threshold
  • Generate comparison report

• Report Generator

  • Create utils/report_generator.py (1195 lines)
  • Generate HTML reports with embedded CSS/JS
  • Create latency distribution charts (Chart.js)
  • Create throughput timeline charts
  • Add metric tables with color-coded deltas
  • Include system information (CPU, memory, DB)
  • Add test configuration summary
  • Support multi-scenario comparison
  • Auto-open in browser

• Docker Compose Generator

  • Create utils/generate_docker_compose.py (423 lines)
  • Generate docker-compose.yaml dynamically
  • Support 1-100 MCP servers
  • Mix SQLite and PostgreSQL backends
  • Assign unique ports to each server
  • Configure resource limits (CPU, memory)
  • Add health checks for all services
  • Include gateway, database, Redis services

Phase 4: Test Orchestration ✅

• Configurable Test Runner

  • Create run-configurable.sh (435 lines)
  • Support --profile argument (light|medium|heavy)
  • Support --scenarios argument (comma-separated)
  • Support --baseline argument (save|compare|skip)
  • Load profile environment variables
  • Execute selected scenarios in sequence
  • Collect all results
  • Generate unified HTML report
  • Exit with code based on regression detection

• Advanced Test Runner

  • Create run-advanced.sh (381 lines)
  • Support --servers argument (1-100)
  • Support --database argument (sqlite|postgresql|both)
  • Generate docker-compose.yaml on-the-fly
  • Start all containers with docker-compose up
  • Wait for services to be healthy
  • Register all servers with gateway
  • Run full test suite
  • Cleanup containers after tests

• All-Scenarios Runner

  • Create run-all.sh (271 lines)
  • Run all profiles sequentially (light → medium → heavy)
  • Run all scenarios for each profile
  • Compare each run against profile baseline
  • Generate master report comparing all profiles
  • Flag critical regressions
  • Provide summary statistics

Phase 5: Documentation ✅

• README.md

  • Create comprehensive guide (309 lines)
  • Add quick start section
  • Document all Makefile targets
  • Explain profile system
  • Show example commands
  • Add troubleshooting section
  • Include expected performance baselines

• MANUAL_TESTING.md

  • Create CLI testing guide (458 lines)
  • Add curl commands for all endpoints
  • Include jq filters for response parsing
  • Add hey benchmarking examples
  • Provide quick smoke test script
  • Add continuous monitoring script
  • Include troubleshooting commands

• PERFORMANCE_STRATEGY.md

  • Create detailed strategy doc (2116 lines)
  • Explain testing philosophy
  • Document baseline management
  • Describe regression detection algorithm
  • Provide performance tuning tips
  • Document environment optimization (.env.example updates)
  • Include CI/CD integration examples

• Configuration Updates

  • Update .env.example with performance-optimized defaults
  • Add LOG_LEVEL=ERROR for testing
  • Add DISABLE_ACCESS_LOG=true
  • Add MCPGATEWAY_LOGGING_OPTIMIZED=true
  • Document performance impact of settings

Phase 6: Test Payloads ✅

• Tools Payloads

  • Create payloads/tools/list_tools.json
  • Create payloads/tools/get_system_time.json
  • Create payloads/tools/convert_time.json
  • Add parameter variations for testing

• Resources Payloads

  • Create payloads/resources/list_resources.json
  • Create payloads/resources/read_timezone_info.json
  • Create payloads/resources/read_world_times.json
  • Test StreamableHTTP resources

• Prompts Payloads

  • Create payloads/prompts/list_prompts.json
  • Create payloads/prompts/get_compare_timezones.json
  • Add argument templates

Phase 7: Integration & Bug Fixes ✅

• Bug Fix: JWT Team Dict Handling

  • Fix token_scoping.py team ID extraction (lines 373-375)
  • Handle both dict and string team formats
  • Normalize token_teams in _check_resource_team_ownership()
  • Maintain backward compatibility
  • Test with real JWT tokens

• Bug Fix: LoggingService Performance

  • Add MCPGATEWAY_LOGGING_OPTIMIZED setting
  • Skip expensive stack inspection when optimized=true
  • Reduce logging overhead from 50% to <5%
  • Achieve 251x performance improvement (7 → 1810 RPS)

• Bug Fix: Ctrl+C Handling

  • Add signal handlers to benchmark scripts
  • Cleanup background processes on interrupt
  • Graceful shutdown of hey processes
  • Prevent zombie processes

• Code Quality

  • Fix all pylint issues (10.00/10 rating)
  • Add pylint disable comments for acceptable patterns
  • Convert strings to f-strings in support_bundle_service.py
  • Add import-outside-toplevel comments for optional imports

Phase 8: Testing & Validation ✅

• Functional Testing

  • Test Makefile targets individually
  • Validate profile loading
  • Test authentication flow
  • Verify service health checks
  • Test baseline save/load/compare
  • Validate report generation

• Performance Validation

  • Run light profile and verify metrics
  • Run medium profile and check scalability
  • Run heavy profile and validate stability
  • Test with 4 MCP servers (SQLite + PostgreSQL)
  • Verify 251x improvement sustained

• Documentation Validation

  • Verify all README examples work
  • Test all MANUAL_TESTING.md commands
  • Validate PERFORMANCE_STRATEGY.md accuracy
  • Check .env.example settings

✅ Success Criteria

• Automated Testing: Complete Makefile with 40+ targets for test automation
• Scenario Coverage: 6 benchmark scenarios covering all core API operations
• Profile System: 3 load profiles (light, medium, heavy) with configurable parameters
• Baseline Management: Automatic baseline storage, comparison, and regression detection
• Reporting: HTML reports with charts, metrics, and color-coded deltas
• Multi-Server Testing: Docker Compose generation for 1-100 MCP servers
• Documentation: 3 comprehensive docs (README, MANUAL_TESTING, PERFORMANCE_STRATEGY)
• Utilities: 6 helper scripts/tools for automation (setup-auth, check-services, baseline_manager, compare_results, report_generator, generate_docker_compose)
• Performance: 251x improvement verified (7 → 1810 RPS with optimizations)
• Bug Fixes: JWT team handling, logging performance, Ctrl+C handling all resolved
• Code Quality: Pylint 10.00/10 rating maintained

📊 Performance Impact

Before Performance Testing Framework

• No systematic benchmarking
• No regression detection
• Unknown performance baselines
• Manual testing only
• No load profile definitions
• No baseline comparison

After Performance Testing Framework

Key Optimizations Enabled

1. Logging Performance: MCPGATEWAY_LOGGING_OPTIMIZED=true (251x improvement)
2. Access Log Disabling: DISABLE_ACCESS_LOG=true (reduces I/O overhead)
3. Log Level Tuning: LOG_LEVEL=ERROR for testing (minimal logging)
4. Database Optimization: Identified query bottlenecks via database-benchmark.sh
5. Connection Pooling: Validated pool sizing via mixed-workload.sh

📝 Additional Notes

🔹 Baseline-Driven Development: All performance changes validated against baselines before merge, ensuring no regressions slip into production.

🔹 Profile-Based Testing: Light (developer laptop), Medium (CI), Heavy (production simulation) profiles enable testing across deployment scenarios.

🔹 Hey Tool Integration: Using hey (Apache Bench successor) for HTTP load testing provides detailed latency distributions and throughput metrics.

🔹 Docker Compose Automation: Dynamic generation of multi-server test environments enables scalability testing without manual configuration.

🔹 Regression Thresholds: Configurable thresholds (default 10% degradation) balance sensitivity with real-world variance.

🔹 HTML Reporting: Interactive reports with Chart.js visualizations make performance data accessible to non-technical stakeholders.

🔹 CI/CD Ready: All scripts designed for integration with GitHub Actions, GitLab CI, Jenkins, etc.

🔹 Future Extensions:

• WebSocket performance testing
• SSE streaming benchmarks
• Database migration performance
• Plugin overhead measurement
• Multi-region latency testing
• Chaos engineering integration

🏁 Definition of Done

• All implementation tasks completed (Phases 1-8)
• 40+ Makefile targets operational
• 6 benchmark scenarios functional
• 3 load profiles defined and tested
• Baseline management system working
• Automated regression detection active
• HTML report generation functional
• All documentation complete (3 docs, 2883 lines)
• All bug fixes merged (JWT, logging, Ctrl+C)
• Code quality verified (pylint 10.00/10)
• 251x performance improvement sustained
• Docker multi-server testing validated
• .env.example updated with optimizations
• Manual testing guide validated
• Team review completed and approved

🎯 Metrics Summary

Project Impact: 8,343 insertions, 57 deletions, 42 files changed