Consider the following Gherkin scenario:
Scenario Outline: Add two numbers
Given I have entered <number1> into the calculator
And I have entered <number2> into the calculator
When I press add
Then the result should be <number3> on the screen
Examples:
| number1 | number2 | number3 |
| 1 | 2 | 3 |
| 1 | 3 | 4 |
SpecFlow will generate the unit test names like this:
AddTwoNumbers_Variant0
AddTwoNumbers_Variant1
The reason for this is that the numbers in the first column are the same, and generated unit test names of course need to be unique. So even if there is only a single duplicate value in that first column, test names will be generated ending with Variant[incrementing number here].
If this is the case, the VsTestScenarioOutlineExampleMatcher in Pickles will not be able to match the examples of the Scenario Outline with the unit test result. It expects the test name to end in the value of that first column. This leads to all tests of the Scenario Outline being marked as Inconclusive.
I've been looking into this and unfortunately, the .trx test results file generated by VsTest is way less useable than MsTest's in this respect, where matching is much easier because of extra information in the results file. This is how VsTest's trx file looks like. I've removed things like GUID's etc. which are not useable, which means there isn't a lot to work with:
<Results>
<UnitTestResult testName="AddTwoNumbers_Variant0" outcome="Passed">
<Output>
<StdOut>-> Using app.config
Given I have entered 1 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(1) (0.0s)
And I have entered 2 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(2) (0.0s)
When I press add
-> done: StepDefinition1.WhenIPressAdd() (0.0s)
Then the result should be 3 on the screen
-> done: StepDefinition1.ThenTheResultShouldBeOnTheScreen(3) (0.0s)
</StdOut>
</Output>
</UnitTestResult>
</Results>
<TestDefinitions>
<UnitTest name="AddTwoNumbers_Variant0">
<TestCategory>
<TestCategoryItem TestCategory="mytag" />
</TestCategory>
<TestMethod name="AddTwoNumbers_Variant0" />
</UnitTest>
</TestDefinitions>
I can't come up with a good solution for this at the moment, so I hope one of you might have a great idea. I'd be happy to build it! Looking forward to your ideas and thoughts.
Consider the following Gherkin scenario:
SpecFlow will generate the unit test names like this:
AddTwoNumbers_Variant0AddTwoNumbers_Variant1The reason for this is that the numbers in the first column are the same, and generated unit test names of course need to be unique. So even if there is only a single duplicate value in that first column, test names will be generated ending with
Variant[incrementing number here].If this is the case, the
VsTestScenarioOutlineExampleMatcherin Pickles will not be able to match the examples of the Scenario Outline with the unit test result. It expects the test name to end in the value of that first column. This leads to all tests of the Scenario Outline being marked as Inconclusive.I've been looking into this and unfortunately, the .trx test results file generated by VsTest is way less useable than MsTest's in this respect, where matching is much easier because of extra information in the results file. This is how VsTest's trx file looks like. I've removed things like GUID's etc. which are not useable, which means there isn't a lot to work with:
I can't come up with a good solution for this at the moment, so I hope one of you might have a great idea. I'd be happy to build it! Looking forward to your ideas and thoughts.