Skip to content

Test cases of Scenario Outline not always matched with results when using VsTest #507

@rik-smeets

Description

@rik-smeets

Consider the following Gherkin scenario:

Scenario Outline: Add two numbers
	Given I have entered <number1> into the calculator
	And I have entered <number2> into the calculator
	When I press add
	Then the result should be <number3> on the screen

	Examples: 
	| number1 | number2 | number3 |
	| 1       | 2       | 3       |
	| 1       | 3       | 4       | 

SpecFlow will generate the unit test names like this:

  • AddTwoNumbers_Variant0
  • AddTwoNumbers_Variant1

The reason for this is that the numbers in the first column are the same, and generated unit test names of course need to be unique. So even if there is only a single duplicate value in that first column, test names will be generated ending with Variant[incrementing number here].

If this is the case, the VsTestScenarioOutlineExampleMatcher in Pickles will not be able to match the examples of the Scenario Outline with the unit test result. It expects the test name to end in the value of that first column. This leads to all tests of the Scenario Outline being marked as Inconclusive.

I've been looking into this and unfortunately, the .trx test results file generated by VsTest is way less useable than MsTest's in this respect, where matching is much easier because of extra information in the results file. This is how VsTest's trx file looks like. I've removed things like GUID's etc. which are not useable, which means there isn't a lot to work with:

  <Results>
      <UnitTestResult testName="AddTwoNumbers_Variant0" outcome="Passed">
      <Output>
        <StdOut>-> Using app.config
Given I have entered 1 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(1) (0.0s)
And I have entered 2 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(2) (0.0s)
When I press add
-> done: StepDefinition1.WhenIPressAdd() (0.0s)
Then the result should be 3 on the screen
-> done: StepDefinition1.ThenTheResultShouldBeOnTheScreen(3) (0.0s)
         </StdOut>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="AddTwoNumbers_Variant0">
      <TestCategory>
        <TestCategoryItem TestCategory="mytag" />
      </TestCategory>
      <TestMethod name="AddTwoNumbers_Variant0" />
    </UnitTest>
  </TestDefinitions>

I can't come up with a good solution for this at the moment, so I hope one of you might have a great idea. I'd be happy to build it! Looking forward to your ideas and thoughts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions