Skip to content

test: Add MC/DC tests for loop pattern detector (stuck_detector)#11600

Merged
enyst merged 10 commits intoOpenHands:mainfrom
phenric26:feature/mcdc-stuck-detector-tests
Dec 29, 2025
Merged

test: Add MC/DC tests for loop pattern detector (stuck_detector)#11600
enyst merged 10 commits intoOpenHands:mainfrom
phenric26:feature/mcdc-stuck-detector-tests

Conversation

@phenric26
Copy link
Copy Markdown
Contributor

Summary of PR

This PR enhances the robustness and test coverage of the StuckDetector by adding 6 new unit test cases for the _is_stuck_action_observation_pattern method.

Why this is needed: The existing test for this method only covered the "happy path" (the A-B-A-B pattern being successfully detected). The complex boolean logic in the D1 (len_check) and D2 (actions_equal) decisions was not being tested for independent failures.

These new tests implement Modified Condition/Decision Coverage (MC/DC), ensuring that each condition (CD1 through CD6) is shown to independently affect the decision's outcome.

Summary of Changes:

Adds 6 new tests to tests/unit/controller/test_stuck.py.

The tests validate the 3 MC/DC cases for the D1 Decision (length check).

The tests validate the 5 MC/DC cases for the D2 Decision (actions_equal), covering all pattern-breaking failures (e.g., test_fail_pattern_break_A6_A4).

Adds reusable event Mocks (Ap, Ai, Adp, etc.) to the top of the test file for readability.

Adds a lightweight fixture (stuck_detector_mcdc) to isolate these tests.

Result: All tests (new and old) are passing. The test count in the test_stuck.py file has increased from 22 to 28, confirming the robustness of the loop detection logic.

Screenshot from 2025-10-31 01-19-50

Change Type

  • Bug fix
  • New feature
  • Breaking change
  • Refactor
  • Other (dependency update, docs, typo fixes, etc.)

Checklist

  • I have read and reviewed the code and I understand what the code is doing.
  • I have tested the code to the best of my ability and ensured it works as expected.

Fixes

Resolves #11599

Comment thread tests/unit/controller/test_is_stuck.py Outdated
Comment thread .gitignore Outdated
Copy link
Copy Markdown
Collaborator

@enyst enyst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is very helpful!

@mamoodi
Copy link
Copy Markdown
Collaborator

mamoodi commented Nov 11, 2025

@phenric26 any interest in addressing the comments left by enyst?

@enyst
Copy link
Copy Markdown
Collaborator

enyst commented Nov 16, 2025

@OpenHands Understand the goal of this PR.

  1. set up the project with make build, then run pre-commit on this PR and fix the issues. Note that the PR is from a fork branch phenric26:feature/mcdc-stuck-detector-tests , so handle the remote correctly.
  2. Fix the tests inline with existing project patterns like top-level imports, not inline, as much as possible, and check and dedup these tests with the existing tests on stuck detector (find them)

@openhands-ai
Copy link
Copy Markdown

openhands-ai Bot commented Nov 16, 2025

I'm on it! enyst can track my progress at all-hands.dev

… lint

- Move mocks and imports to module top-level
- Deduplicate with existing pattern test and keep consistent naming
- Format per ruff

Co-authored-by: openhands <openhands@all-hands.dev>
@enyst enyst force-pushed the feature/mcdc-stuck-detector-tests branch from 1a2ed86 to 4ce47d0 Compare November 24, 2025 20:45
enyst and others added 2 commits November 24, 2025 20:48
- Translate non-English comments to English

- Replace terse test symbols (Ap/Ai/etc.) with descriptive names

- Use action_x/obs_x phrasing in inline comments

- Prune redundant index-variant tests; keep guards and ignore-behavior tests

- Focus on _is_stuck_action_observation_pattern behavior

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig neubig requested a review from enyst December 28, 2025 13:40
@neubig
Copy link
Copy Markdown
Contributor

neubig commented Dec 28, 2025

Hey @enyst , this has been open for a while, if you're reviewing could you take another look?

Comment thread .gitignore Outdated
enyst and others added 3 commits December 29, 2025 12:41
- Fix D205: Add blank line between summary line and description in docstring
- Remove extra blank lines between test methods (E303)

Co-authored-by: openhands <openhands@all-hands.dev>
Copy link
Copy Markdown
Collaborator

@enyst enyst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@enyst enyst merged commit a3e85e2 into OpenHands:main Dec 29, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve test coverage for _is_stuck_action_observation_pattern with MC/DC

4 participants