What is Mutation Testing?

Mutation testing evaluates the quality of your test suite by introducing small changes (mutations) to the source code and checking if the tests detect them.

How it works

1. A mutation tool creates small variations called mutants:

   • Changing > to >= or <
   • Replacing true with false
   • Removing a function call
   • Changing + to -
   • Replacing a string with an empty string

2. For each mutant, the tool runs your test suite

3. If tests fail → mutant killed ✅ (tests caught the change)

4. If tests pass → mutant survived ❌ (tests missed the change)

A high mutation score (% of killed mutants) means your tests are effective. A low score means bugs could hide undetected.

Why it matters

Code coverage tells you which lines were executed — not whether tests actually verify correct behavior. Mutation testing answers: "If a bug were introduced here, would our tests catch it?"

Concrete examples from this codebase

Current State

• 15 source files in lib/
• 2 test files: test/widget_test.dart, test/features/match/models/match_test.dart
• No CI pipeline yet
• Key modules untested: nostr_service.dart, key_manager.dart, relay_config_provider.dart, all providers

Recommended Tool: DCM (Dart Code Metrics)

DCM is the only actively maintained tool that supports mutation testing for Dart/Flutter. It includes:

• ✅ Dart/Flutter first-class support
• ✅ Mutation testing built-in (dcm run-mutations)
• ✅ Multiple mutators (arithmetic, boolean, boundary, removal, etc.)
• ✅ HTML/JSON reports
• ✅ CI integration
• ✅ Configurable rules and exclusions

Alternatives Considered

Implementation Plan

Phase 1: Setup and Baseline

1. Install DCM:

   dart pub global activate dcm

2. Add analysis_options.yaml configuration:

   dart_code_metrics:
     mutations:
       mutators:
         - arithmetic
         - boolean_literal
         - conditional_boundary
         - remove_statement
         - negate_conditional
       exclude:
         - "lib/main.dart"
         - "lib/shared/theme/**"
         - "**/*.g.dart"
         - "**/*.freezed.dart"

3. Run baseline:

   dcm run-mutations lib/ --report-format=html

Phase 2: Expand Test Coverage for Key Modules

Priority order for writing tests (based on criticality):

Phase 3: CI Integration

Create .github/workflows/mutation-testing.yaml:

name: Mutation Testing

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * 1'  # Weekly Monday 6:00 UTC
  workflow_dispatch:

jobs:
  mutation-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: subosito/flutter-action@v2
        with:
          channel: stable

      - run: flutter pub get
      - run: dart pub global activate dcm

      - name: Run mutation tests
        continue-on-error: true  # Non-blocking initially
        run: dcm run-mutations lib/ --report-format=json --output=mutation-report.json

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: mutation-report
          path: mutation-report.json
          retention-days: 30

Acceptance Criteria

• DCM installed and configured
• Baseline mutation score measured
• Tests written for surviving mutants in critical modules (match logic, Nostr events)
• CI workflow running weekly (non-blocking)
• Threshold set based on baseline (suggest starting at 50%)

References

• DCM Mutation Testing
• Mutation Testing Concepts
• MostroP2P/mostro#618 — Same initiative for Rust
• lnp2pBot/bot#760 — Same initiative for TypeScript