feat: add mutation testing with mutation_test package#39
Conversation
Add mutation testing infrastructure to evaluate test suite quality: - Add mutation_test ^1.8.0 to dev_dependencies - Create mutation_test.xml configuration targeting match models - Add docs/ci/mutation-testing.yaml (CI workflow template — copy to .github/workflows/ to activate) - Add docs/MUTATION_TESTING.md with usage guide - Add mutation-test-report/ to .gitignore The mutation testing starts with lib/features/match/models/ which has the most comprehensive tests. As test coverage grows, more directories can be added to the configuration. CI jobs are non-blocking initially (continue-on-error: true). Set the failure threshold in mutation_test.xml once a baseline is established. Note: The GitHub App cannot push workflow files directly. The CI workflow is provided in docs/ci/ and must be manually copied to .github/workflows/ by a maintainer. Closes #37
Use pattern attribute on <regex> elements as required by mutation_test package. Fix widget tests to wrap ChokeApp in ProviderScope and update match status expectations to use kebab-case serialization format.
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR introduces comprehensive mutation testing infrastructure to evaluate test suite quality. It adds configuration files (mutation_test.xml, GitHub Actions workflow), documentation, a dart mutation_test dev dependency, updates .gitignore, and adjusts test expectations to align with new JSON serialization formats. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/ci/mutation-testing.yaml`:
- Around line 3-10: The workflow is missing a pull_request trigger so the job
mutation-test-pr (which guards with if: github.event_name == 'pull_request') is
never executed; update the workflow's on: block to include a pull_request
trigger (restrict to same branches as push, e.g., branches: [main]) so PR events
will run the workflow and make the mutation-test-pr job reachable.
In `@docs/MUTATION_TESTING.md`:
- Around line 53-57: Update the docs/MUTATION_TESTING.md text to explicitly
instruct readers to copy the workflow template from docs/ci/ into the repository
workflows folder so it becomes active: add a step telling users to copy or move
the file (e.g., "copy docs/ci/mutation-testing.yaml to
.github/workflows/mutation-testing.yaml") and mention that the workflow will
only run once placed under .github/workflows/, referencing the workflow name
`.github/workflows/mutation-testing.yaml` so it's clear the file in docs/ci/ is
a template not the active workflow.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
.gitignoredocs/MUTATION_TESTING.mddocs/ci/mutation-testing.yamlmutation_test.xmlpubspec.yamltest/features/match/models/match_test.darttest/widget_test.dart
| on: | ||
| push: | ||
| branches: [main] | ||
| schedule: | ||
| # Weekly on Monday 6:00 UTC | ||
| - cron: '0 6 * * 1' | ||
| workflow_dispatch: | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="docs/ci/mutation-testing.yaml"
# Check if file exists
if [ ! -f "$FILE" ]; then
echo "File not found: $FILE"
exit 1
fi
echo "=== File: docs/ci/mutation-testing.yaml ==="
echo
echo "=== Lines 1-20 (trigger block) ==="
head -20 "$FILE"
echo
echo "=== Lines 40-55 (checking for PR job guard) ==="
sed -n '40,55p' "$FILE"
echo
echo "=== Checking for pull_request trigger ==="
if grep -n "pull_request:" "$FILE"; then
echo "✓ pull_request trigger FOUND"
else
echo "✗ pull_request trigger NOT FOUND"
fi
echo
echo "=== Checking for PR-gated job ==="
grep -n "if:.*pull_request" "$FILE" || trueRepository: grunch/choke
Length of output: 1070
Add pull_request trigger — mutation-test-pr job is unreachable.
The workflow defines a mutation-test-pr job at line 47 with if: github.event_name == 'pull_request', but the trigger block (lines 3–10) lacks a pull_request trigger. The workflow will never run on PRs, making that job unreachable dead code.
🔧 Proposed fix
on:
push:
branches: [main]
+ pull_request:
+ branches: [main]
schedule:
# Weekly on Monday 6:00 UTC
- cron: '0 6 * * 1'
workflow_dispatch:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| on: | |
| push: | |
| branches: [main] | |
| schedule: | |
| # Weekly on Monday 6:00 UTC | |
| - cron: '0 6 * * 1' | |
| workflow_dispatch: | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| branches: [main] | |
| schedule: | |
| # Weekly on Monday 6:00 UTC | |
| - cron: '0 6 * * 1' | |
| workflow_dispatch: | |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/ci/mutation-testing.yaml` around lines 3 - 10, The workflow is missing a
pull_request trigger so the job mutation-test-pr (which guards with if:
github.event_name == 'pull_request') is never executed; update the workflow's
on: block to include a pull_request trigger (restrict to same branches as push,
e.g., branches: [main]) so PR events will run the workflow and make the
mutation-test-pr job reachable.
There was a problem hiding this comment.
@coderabbitai[bot] Resolved in commit 75de743. ✅
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
| The GitHub Actions workflow (`.github/workflows/mutation-testing.yaml`) runs: | ||
|
|
||
| 1. **Full baseline** on pushes to `main` and weekly (Monday 6:00 UTC) | ||
| 2. **Incremental** on PRs (only changed source files) | ||
|
|
There was a problem hiding this comment.
Clarify activation step for the workflow path.
This section implies the workflow is already active at .github/workflows/..., but the file introduced here is under docs/ci/. Add an explicit “copy template to .github/workflows/” step to avoid confusion.
📝 Suggested wording tweak
- The GitHub Actions workflow (`.github/workflows/mutation-testing.yaml`) runs:
+ After copying `docs/ci/mutation-testing.yaml` to `.github/workflows/mutation-testing.yaml`,
+ the GitHub Actions workflow runs:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| The GitHub Actions workflow (`.github/workflows/mutation-testing.yaml`) runs: | |
| 1. **Full baseline** on pushes to `main` and weekly (Monday 6:00 UTC) | |
| 2. **Incremental** on PRs (only changed source files) | |
| After copying `docs/ci/mutation-testing.yaml` to `.github/workflows/mutation-testing.yaml`, | |
| the GitHub Actions workflow runs: | |
| 1. **Full baseline** on pushes to `main` and weekly (Monday 6:00 UTC) | |
| 2. **Incremental** on PRs (only changed source files) | |
🧰 Tools
🪛 LanguageTool
[uncategorized] ~53-~53: The official name of this software platform is spelled with a capital “H”.
Context: ...tegration The GitHub Actions workflow (.github/workflows/mutation-testing.yaml) runs:...
(GITHUB)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/MUTATION_TESTING.md` around lines 53 - 57, Update the
docs/MUTATION_TESTING.md text to explicitly instruct readers to copy the
workflow template from docs/ci/ into the repository workflows folder so it
becomes active: add a step telling users to copy or move the file (e.g., "copy
docs/ci/mutation-testing.yaml to .github/workflows/mutation-testing.yaml") and
mention that the workflow will only run once placed under .github/workflows/,
referencing the workflow name `.github/workflows/mutation-testing.yaml` so it's
clear the file in docs/ci/ is a template not the active workflow.
There was a problem hiding this comment.
@coderabbitai[bot] Resolved in commit 75de743. ✅
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
- docs/ci/mutation-testing.yaml: Add pull_request trigger for mutation-test-pr job - docs/MUTATION_TESTING.md: Clarify that template must be copied to .github/workflows/ - Fixes CodeRabbit review comments on PR #39
There was a problem hiding this comment.
Fixed in commit 75de743. @coderabbitai[bot] marking this as resolved.
There was a problem hiding this comment.
Fixed in commit 75de743. @coderabbitai[bot] marking this as resolved.
There was a problem hiding this comment.
Fixed in commit 75de743. @coderabbitai[bot] marking this as resolved.
Changes Made:
- Added
pull_requesttrigger todocs/ci/mutation-testing.yaml - Clarified workflow activation in `docs/MUTATION_TESTING.md
- Change from 'open' command only to explicit commands for all OS - Add note about manual browser opening as alternative
- Change index.html to mutation-test-report.html (actual filename)
Summary
Adds mutation testing infrastructure to Choke using the
mutation_testpackage (v1.8.0+), the only actively maintained mutation testing tool for Dart/Flutter.Closes #37
What's included
1.
mutation_test.xml— Configurationlib/features/match/models/(the module with the most comprehensive tests)flutter testas the test command (120s timeout)failure="0"(report-only mode)2.
docs/ci/mutation-testing.yaml— CI Workflow TemplateTwo jobs:
mutation-test: Full baseline on pushes tomain+ weekly Monday 6:00 UTCmutation-test-pr: Incremental — only mutates changed source files on PRsBoth are non-blocking (
continue-on-error: true). Reports uploaded as artifacts.3.
docs/MUTATION_TESTING.md— Usage Guide4.
pubspec.yaml— DependencyAdded
mutation_test: ^1.8.0to dev_dependencies.How to use
Reports are generated as HTML in
mutation-test-report/(gitignored).Why mutation_test?
Next steps
mutation_test.xmlas test coverage growsfailurethreshold once baseline is stable.github/workflows/Summary by CodeRabbit
Documentation
Chores