-
-
Notifications
You must be signed in to change notification settings - Fork 616
refactor: convert rerun-only tests to use fixtures #1764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: convert rerun-only tests to use fixtures #1764
Conversation
features/rerun-only.feature
Outdated
| Scenario: Run one feature with 2 failed and 3 passing scenarios | ||
| When I run "behat --no-colors -f progress features/apples.feature" | ||
| When I run "behat features/apples.feature" | ||
| Then it should fail with: | ||
| """ | ||
| ..F.............F.... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@carlos-granados sorry to keep picking on these rerun features.
I think the process you're working through is very helpfully showing where the noise of the setup in the past has made it hard to reason about what the features are actually covering.
I've realised this first scenario is IMO not a scenario at all. It is not testing anything about the rerun feature, only that the fixture we have created fails as expected.
This is actually a precondition of the later scenarios (where we start with the same `Given I run "behat features/apples.feature" but don't assert anything about the result).
The later scenarios assume that if we've run the same steps to setup the fixture, it will fail in the same way as this scenario did - but this is not actually guaranteed (there could have been some filesystem issue for example).
I think the content of this scenario should either be inlined as the first steps of the actual scenarios, or a Background if it is consistent for all scenarios (it isn't in this case).
This would also apply to the feature we looked at the other day, and to #1766
features/rerun-only.feature
Outdated
| 7 steps (5 passed, 2 failed) | ||
| """ | ||
|
|
||
| Scenario: Fixing scenario removes it from the rerun log |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this scenario isn't named quite right for what it covers. Run nothing if there were no previous failures or something would make more sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or maybe this is actually two separate scenarios:
- If Behat has never run before, nothing runs (even with the unchanged
apples.feature- Behat passes because that failing scenario is never executed). - If Behat has run & failed, then was fixed, then
--rerun-onlypassed, then the next run does not run any scenarios.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree that this is better defined as two separate scenarios, I have updated the code
c396653 to
cbc529a
Compare
cbc529a to
21c2836
Compare
21c2836 to
a26568d
Compare
acoulton
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @carlos-granados that's a great improvement
In this case we have re-used the fixtures that we had for the
rerun.featuretests as they were almost identical, we have just added a new feature because the behaviour when we were fixing the tests was slightly different