Context

Mostro Mobile is a Flutter application with critical security requirements for Bitcoin P2P trading. High-quality tests are essential for ensuring the safety of user funds and the reliability of the trading platform.

What is Mutation Testing?

Mutation testing is a technique to measure the quality and effectiveness of our test suite. It works by:

1. Creating mutants: The tool makes small, controlled changes (mutations) to the source code

   • Example: Change == to !=, + to -, true to false
   • Example: Remove a line of code, change a condition boundary
   • Example: Flip boolean conditions

2. Running the test suite: Tests are executed against each mutated version

3. Measuring survival rate:

   • ✅ Mutant killed: Test failed → Good! Tests detected the artificial bug
   • ❌ Mutant survived: Test passed → Bad! Tests did not catch the change

Mutation Score

Score = (Mutants killed / Total mutants) × 100

• > 80%: Excellent test coverage
• 50-80%: Acceptable, room for improvement
• < 50%: Tests are not effectively verifying behavior

Advantages for Mostro Mobile

1. Security-critical code: Trading and key management code needs rigorous verification
2. Detect weak tests: Find tests that "cover" code but do not actually verify correctness
3. Force better assertions: Encourages specific, strict assertions instead of generic ones
4. Find edge cases: Surviving mutants often reveal untested boundary conditions
5. CI integration: Can fail builds if mutation score drops below threshold
6. Documentation: Living documentation of what behavior is actually tested

Implementation Plan

Phase 1: Setup

1. Install mutant package

   dart pub add --dev mutant

2. Configure mutation testing
   Create mutant.yaml:

   mutant:
     targets:
       - lib/
     exclude:
       - "**/*.g.dart"
       - "**/generated/**"
       - "**/*.freezed.dart"
     rules:
       - equality
       - relational
       - unary
       - conditional_boundary
       - literal
       - method_call
     threshold: 60  # Start conservative, increase gradually

Phase 2: CI Integration

Add to .github/workflows/flutter.yml:

- name: Run tests
  run: flutter test

- name: Mutation Testing
  run: dart run mutant
  continue-on-error: true  # Report only initially

- name: Upload mutation report
  uses: actions/upload-artifact@v4
  with:
    name: mutation-report
    path: mutation-report/

Phase 3: Gradual Improvement

• Measure current baseline mutation score
• Set threshold to current score (do not decrease)
• Identify and fix surviving mutants in critical areas:
  • Key management (lib/services/key_management/)
  • Nostr service (lib/services/nostr/)
  • Trade logic (lib/features/trade/)
• Increase threshold by 5% each sprint
• Target: 80% mutation score for security-critical modules

Acceptance Criteria

• Add mutant to dev_dependencies in pubspec.yaml
• Create mutant.yaml configuration
• Add mutation testing step to CI workflow
• Generate baseline mutation report
• Document mutation score interpretation in CONTRIBUTING.md
• Set initial threshold and enforce in CI (can be lenient initially)

Related

• Similar issue in Choke project: Add mutation testing with DCM to evaluate test suite quality grunch/choke#37
• Mostro uses NIP-59 gift wrap, BIP-32 HD keys, and Lightning Network - all areas where mutation testing adds significant value

References

• mutant package on pub.dev
• Mutation Testing Concepts
• Stryker - Industry standard mutation testing framework