[TESTING][DEVOPS] E2E Setup Script Testing with Init Containers

Goal

Create automated end-to-end tests for the setup scripts (ubuntu-contextforge-setup-script.sh, rocky-contextforge-setup-script.sh) using Docker init containers with systemd, validating complete installation across multiple Linux distributions and configurations.

Why Now?

The setup scripts are critical for user onboarding and deployment. Currently:

• No automated testing exists for setup scripts
• Manual testing is time-consuming and error-prone
• Changes may break installations on untested distributions
• PR chore-2193: add Rocky Linux setup script #2490 added significant improvements that need validation matrix

User Stories

Feature: Setup Script Testing

  Scenario: Test on Rocky Linux 9
    Given a fresh Rocky Linux 9 init container
    When I run the setup script with -y --remove-podman --skip-start
    Then Docker should be installed and running
    And uv should be installed
    And the repository should be cloned
    And .env file should be created

  Scenario: Test on Ubuntu 24.04
    Given a fresh Ubuntu 24.04 init container
    When I run the setup script with -y --skip-start
    Then Docker should be installed and running
    And all components should be configured

  Scenario: Test non-interactive mode
    Given any supported distribution
    When I run with -y flag
    Then no prompts should appear
    And script should complete or fail with clear error

Feature: CI Integration

  Scenario: PR modifying setup scripts
    Given a PR that modifies scripts/*-setup-script.sh
    When CI runs
    Then setup script tests should execute
    And results should be reported for each distribution

  Scenario: Matrix testing
    Given the test matrix includes multiple distributions
    When tests run in parallel
    Then each distribution should be tested independently
    And failures should identify the specific distribution

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                    E2E Setup Script Test Architecture                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Makefile Targets:                                                          │
│  ┌────────────────────────────────────────────────────────────────────┐    │
│  │ make test-setup-scripts      # Run all distro tests                │    │
│  │ make test-setup-ubuntu       # Ubuntu only                         │    │
│  │ make test-setup-rocky        # Rocky only                          │    │
│  │ make test-setup-debian       # Debian only                         │    │
│  │ make test-setup-alma         # AlmaLinux only                      │    │
│  │ make test-setup-parallel     # All distros in parallel             │    │
│  │ make test-setup-clean        # Cleanup test containers             │    │
│  └────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│  Test Flow:                                                                  │
│  ┌──────────┐    ┌──────────────┐    ┌─────────────┐    ┌────────────┐    │
│  │ Start    │───▶│ Run Setup   │───▶│ Validate    │───▶│ Report     │    │
│  │ Init     │    │ Script      │    │ Components  │    │ Results    │    │
│  │ Container│    │ (-y mode)   │    │ (Docker,uv) │    │            │    │
│  └──────────┘    └──────────────┘    └─────────────┘    └────────────┘    │
│                                                                              │
│  Container Architecture:                                                     │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │                     Docker Host (Privileged)                         │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ │   │
│  │  │ Rocky 9     │  │ Ubuntu 24.04│  │ Debian 12   │  │ AlmaLinux 9│ │   │
│  │  │ (systemd)   │  │ (systemd)   │  │ (systemd)   │  │ (systemd)  │ │   │
│  │  │             │  │             │  │             │  │            │ │   │
│  │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌────────┐ │ │   │
│  │  │ │ Docker  │ │  │ │ Docker  │ │  │ │ Docker  │ │  │ │ Docker │ │ │   │
│  │  │ │ (DinD)  │ │  │ │ (DinD)  │ │  │ │ (DinD)  │ │  │ │ (DinD) │ │ │   │
│  │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └────────┘ │ │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────────┘ │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Distribution Test Matrix

Tier 1: Primary Support (Must Pass)

Tier 2: Secondary Support (Should Pass)

Tier 3: Best Effort

Test Configurations

Configuration Matrix

Validation Checks

# Post-installation validation script
validate_installation() {
    local errors=0
    
    # Check Docker
    if ! command -v docker &>/dev/null; then
        echo "FAIL: Docker not installed"
        ((errors++))
    elif ! docker --version; then
        echo "FAIL: Docker not working"
        ((errors++))
    else
        echo "PASS: Docker $(docker --version)"
    fi
    
    # Check Docker service
    if ! systemctl is-active docker &>/dev/null; then
        echo "FAIL: Docker service not running"
        ((errors++))
    else
        echo "PASS: Docker service active"
    fi
    
    # Check Docker Compose
    if ! docker compose version &>/dev/null; then
        echo "FAIL: Docker Compose not available"
        ((errors++))
    else
        echo "PASS: Docker Compose $(docker compose version --short)"
    fi
    
    # Check uv
    if ! command -v uv &>/dev/null && [[ ! -x "$HOME/.local/bin/uv" ]]; then
        echo "FAIL: uv not installed"
        ((errors++))
    else
        echo "PASS: uv installed"
    fi
    
    # Check repository
    if [[ ! -d "$HOME/mcp-context-forge/.git" ]]; then
        echo "FAIL: Repository not cloned"
        ((errors++))
    else
        echo "PASS: Repository cloned"
    fi
    
    # Check .env
    if [[ ! -f "$HOME/mcp-context-forge/.env" ]]; then
        echo "FAIL: .env not created"
        ((errors++))
    else
        echo "PASS: .env created"
    fi
    
    return $errors
}

Implementation Tasks

Phase 1: Test Infrastructure

• Create tests/e2e/setup-scripts/ directory structure
• Create base test script test-setup-common.sh with shared functions
• Create distribution-specific test wrappers
• Add Makefile targets for individual and parallel testing
• Create validation script for post-installation checks

Phase 2: Init Container Setup

• Create Dockerfile templates for systemd-enabled containers
• Ubuntu systemd container (tests/e2e/setup-scripts/Dockerfile.ubuntu-systemd)
• Debian systemd container (tests/e2e/setup-scripts/Dockerfile.debian-systemd)
• Document Rocky/RHEL using official -init or -ubi-init images
• Test container builds locally

Phase 3: Test Implementation

• Implement Rocky Linux 9 test
• Implement Ubuntu 24.04 test
• Implement Ubuntu 22.04 test
• Implement Debian 12 test
• Implement AlmaLinux 9 test
• Add timeout handling for stuck tests
• Add cleanup on test failure

Phase 4: CI Integration

• Create GitHub Actions workflow for setup script tests
• Configure matrix strategy for parallel distribution testing
• Add workflow trigger on scripts/*-setup-script.sh changes
• Configure artifact upload for test logs
• Add status badges to README

Phase 5: Documentation

• Document how to run tests locally
• Document how to add new distribution tests
• Add troubleshooting guide for common failures
• Update CLAUDE.md with test commands

Proposed Makefile Targets

# =============================================================================
# Setup Script E2E Testing
# =============================================================================

SETUP_TEST_DIR := tests/e2e/setup-scripts
SETUP_TEST_TIMEOUT := 600

## Test setup scripts on all supported distributions
test-setup-scripts: test-setup-rocky test-setup-ubuntu test-setup-debian
	@echo "All setup script tests completed"

## Test Rocky Linux setup script
test-setup-rocky: test-setup-rocky9

## Test Rocky Linux 9
test-setup-rocky9:
	@echo "Testing Rocky Linux 9..."
	@timeout $(SETUP_TEST_TIMEOUT) $(SETUP_TEST_DIR)/test-rocky9.sh

## Test Ubuntu setup script
test-setup-ubuntu: test-setup-ubuntu2404 test-setup-ubuntu2204

## Test Ubuntu 24.04
test-setup-ubuntu2404:
	@echo "Testing Ubuntu 24.04..."
	@timeout $(SETUP_TEST_TIMEOUT) $(SETUP_TEST_DIR)/test-ubuntu2404.sh

## Test Ubuntu 22.04
test-setup-ubuntu2204:
	@echo "Testing Ubuntu 22.04..."
	@timeout $(SETUP_TEST_TIMEOUT) $(SETUP_TEST_DIR)/test-ubuntu2204.sh

## Test Debian setup script
test-setup-debian: test-setup-debian12

## Test Debian 12
test-setup-debian12:
	@echo "Testing Debian 12..."
	@timeout $(SETUP_TEST_TIMEOUT) $(SETUP_TEST_DIR)/test-debian12.sh

## Test AlmaLinux 9
test-setup-alma9:
	@echo "Testing AlmaLinux 9..."
	@timeout $(SETUP_TEST_TIMEOUT) $(SETUP_TEST_DIR)/test-alma9.sh

## Run all setup tests in parallel
test-setup-parallel:
	@echo "Running setup tests in parallel..."
	@parallel --tag --line-buffer ::: \
		"make test-setup-rocky9" \
		"make test-setup-ubuntu2404" \
		"make test-setup-ubuntu2204" \
		"make test-setup-debian12"

## Clean up test containers
test-setup-clean:
	@docker ps -a --filter "name=contextforge-test-" -q | xargs -r docker rm -f
	@docker images --filter "reference=contextforge-test-*" -q | xargs -r docker rmi -f

Example Test Script

#!/bin/bash
# tests/e2e/setup-scripts/test-rocky9.sh

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/test-common.sh"

DISTRO="rocky9"
IMAGE="rockylinux/rockylinux:9-ubi-init"
SCRIPT="rocky-contextforge-setup-script.sh"
CONTAINER_NAME="contextforge-test-${DISTRO}-$$"

main() {
    log_info "Starting $DISTRO setup script test"
    
    # Start init container
    start_init_container "$CONTAINER_NAME" "$IMAGE"
    
    # Setup test user
    setup_test_user "$CONTAINER_NAME"
    
    # Copy and run setup script
    copy_setup_script "$CONTAINER_NAME" "$SCRIPT"
    run_setup_script "$CONTAINER_NAME" "-y --remove-podman --skip-start"
    
    # Validate installation
    validate_installation "$CONTAINER_NAME"
    
    # Cleanup
    cleanup_container "$CONTAINER_NAME"
    
    log_success "$DISTRO test passed!"
}

trap 'cleanup_container "$CONTAINER_NAME" || true' EXIT
main "$@"

GitHub Actions Workflow

# .github/workflows/test-setup-scripts.yml
name: Setup Script Tests

on:
  push:
    paths:
      - 'scripts/*-setup-script.sh'
      - 'tests/e2e/setup-scripts/**'
  pull_request:
    paths:
      - 'scripts/*-setup-script.sh'
      - 'tests/e2e/setup-scripts/**'
  workflow_dispatch:

jobs:
  test-setup-scripts:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        distro:
          - { name: "rocky9", image: "rockylinux/rockylinux:9-ubi-init", script: "rocky" }
          - { name: "ubuntu2404", image: "ubuntu:24.04", script: "ubuntu" }
          - { name: "ubuntu2204", image: "ubuntu:22.04", script: "ubuntu" }
          - { name: "debian12", image: "debian:12", script: "ubuntu" }
          - { name: "alma9", image: "almalinux:9", script: "rocky" }
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Run setup script test
        run: make test-setup-${{ matrix.distro.name }}
        timeout-minutes: 15
      
      - name: Upload logs on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: logs-${{ matrix.distro.name }}
          path: /tmp/contextforge-test-*.log

Success Criteria

• All Tier 1 distributions pass automated tests
• Tests run in under 10 minutes per distribution
• Failed tests provide clear error messages
• Tests are reproducible locally and in CI
• Parallel execution works without conflicts
• Cleanup removes all test artifacts

Definition of Done

• Test infrastructure created in tests/e2e/setup-scripts/
• Makefile targets added and documented
• All Tier 1 distributions passing
• At least 2 Tier 2 distributions passing
• GitHub Actions workflow running on PRs
• Documentation updated with test instructions
• CLAUDE.md updated with test commands

Related Issues

• chore-2193: add Rocky Linux setup script #2490 - Add Rocky Linux setup script (implements the script being tested)
• [CHORE]: Add Rocky Linux setup script variant #2193 - Original Rocky Linux request

References

• Docker systemd in containers
• Rocky Linux container images
• GitHub Actions matrix strategy