[FEATURE][SECURITY]: Audit Logging System

Goal

Implement a tamper-resistant audit logging system that tracks all security-relevant actions, administrative changes, and data access patterns. This provides forensic capabilities, compliance support, and real-time security monitoring.

Why Now?

1. Compliance Requirements: SOC2, GDPR, HIPAA require audit trails
2. Security Forensics: Investigate incidents with detailed logs
3. Insider Threat Detection: Monitor for suspicious access patterns
4. Configuration Tracking: Audit all administrative changes
5. Data Access Monitoring: Track who accessed what data

Status: IMPLEMENTED

The audit logging system has been implemented in mcpgateway/services/audit_trail_service.py. Key components:

Existing Implementation:

• AuditTrail model in db.py
• AuditTrailService class with:
  • AuditAction enum (CREATE, READ, UPDATE, DELETE, EXECUTE, ACCESS, EXPORT, IMPORT)
  • DataClassification levels (PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED)
  • Correlation ID support via get_or_generate_correlation_id()
• Database storage with timestamps

Configuration:

• AUDIT_TRAIL_ENABLED - Enable/disable audit logging
• AUDIT_TRAIL_LEVEL - Log level filtering (see [PERFORMANCE]: Audit Trail Performance & Configuration Enhancements #1745)

📖 User Stories

Scenario: Log server creation
  Given AUDIT_TRAIL_ENABLED=true
  When an admin creates a new MCP server
  Then an audit record should be created
  And it should include: user, action (CREATE), resource_type (server), resource_id
  And it should include a timestamp and correlation_id

Scenario: Log tool deletion
  Given AUDIT_TRAIL_ENABLED=true
  When an admin deletes a tool
  Then an audit record should be created with action DELETE
  And the old values should be preserved in the record

Scenario: Log configuration updates
  Given AUDIT_TRAIL_ENABLED=true
  When an admin updates gateway OAuth settings
  Then an audit record should capture before/after values
  And sensitive fields should be redacted

Scenario: Log resource access
  Given AUDIT_TRAIL_LEVEL=all
  When a user reads a resource
  Then an audit record should be created with action ACCESS
  And it should include the data classification level

Scenario: Log tool execution
  Given AUDIT_TRAIL_ENABLED=true
  When a user executes a tool
  Then an audit record should be created with action EXECUTE
  And it should include execution duration and success/failure

Scenario: Search by user
  Given audit logs exist
  When I query GET /admin/audit?user=alice@example.com
  Then I should see all audit records for that user
  And results should be paginated

Scenario: Search by date range
  Given audit logs exist
  When I query GET /admin/audit?from=2026-01-01&to=2026-01-31
  Then I should see records within that date range

Scenario: Export audit logs
  Given audit logs exist
  When I call GET /admin/audit/export?format=csv
  Then I should receive a CSV file with all matching records

Scenario: Hash chain verification
  Given AUDIT_TRAIL_INTEGRITY_ENABLED=true
  When an audit record is created
  Then it should include a hash of the previous record
  And the chain should be verifiable

Scenario: Detect tampering
  Given a chain of audit records exists
  When the chain integrity is verified
  Then any tampered records should be detected
  And an alert should be raised

🏗 Architecture

Audit Trail Flow

sequenceDiagram
    participant API
    participant Service
    participant AuditService
    participant DB

    API->>Service: Create server
    Service->>Service: Execute operation
    Service->>AuditService: log_action(CREATE, server, data)
    AuditService->>AuditService: Redact sensitive fields
    AuditService->>AuditService: Calculate hash chain
    AuditService->>DB: INSERT audit_trail
    Service-->>API: Response

Audit Trail Data Model

erDiagram
    AUDIT_TRAIL {
        bigint id PK
        uuid event_id UK
        datetime timestamp
        string event_type
        string event_action
        string user_id
        string user_email
        string resource_type
        string resource_id
        json old_values
        json new_values
        string correlation_id
        string classification
        string previous_hash
        string record_hash
    }

📋 Implementation Tasks

Phase 1: Core Service (COMPLETE)

• Create AuditTrailService class
• Define AuditAction enum
• Define DataClassification enum
• Implement basic logging
• Add correlation ID support

Phase 2: Query API

• Add /admin/audit GET endpoint with filtering
• Add pagination support
• Add export endpoint (CSV, JSON)
• Add Admin UI for audit viewing

Phase 3: Integrity Features

• Implement hash chain for records
• Add integrity verification endpoint
• Add tamper detection alerting

Phase 4: Enhanced Logging

• Add before/after value capture
• Implement sensitive field redaction
• Add tool execution logging with metrics
• Add data access classification

Phase 5: Retention & Cleanup

• Implement configurable retention (see [PERFORMANCE]: Audit Trail Performance & Configuration Enhancements #1745)
• Add batch cleanup for old records
• Add archival to external storage

Phase 6: Testing

• Unit tests for audit service
• Integration tests for query API
• Tests for integrity verification
• Performance tests for high volume

⚙️ Configuration Example

# Audit Trail Settings
AUDIT_TRAIL_ENABLED=true
AUDIT_TRAIL_LEVEL=writes_only     # all, writes_only, mutations_only
AUDIT_TRAIL_RETENTION_DAYS=90     # See #1745 for retention config
AUDIT_TRAIL_INTEGRITY_ENABLED=true

# Redacted Fields (never logged)
AUDIT_TRAIL_REDACTED_FIELDS=password,token,secret,api_key

# Data Classification
AUDIT_TRAIL_DEFAULT_CLASSIFICATION=internal

✅ Success Criteria

• Basic audit logging working
• CRUD actions logged
• Query API implemented
• Export functionality working
• Hash chain integrity implemented
• Sensitive data redacted
• Admin UI for audit viewing
• Retention policy enforced

🏁 Definition of Done

• AuditTrailService implemented
• AuditAction and DataClassification enums
• Database model created
• Query API endpoints
• Export functionality
• Integrity verification
• Admin UI integration
• Unit tests pass
• Integration tests pass
• Code passes make verify
• Documentation complete

🔗 Related Issues

• Related: [PERFORMANCE]: Audit Trail Performance & Configuration Enhancements #1745 (Audit Trail Performance & Configuration)
• Related: [FEATURE][SECURITY]: Enhanced session management for admin UI #541 (Session Management - logs session events)
• Existing: mcpgateway/services/audit_trail_service.py
• Existing: mcpgateway/db.py:AuditTrail