Skip to content

Allure-specflow doesn't make distinction between test and product defects #399

@delatrie

Description

@delatrie

I'm submitting a ...

  • bug report
  • feature request
  • support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

Currently, any exception occurred in a step definition makes the step and the test failed (indicated with red in the report) regardless of whether it is a failed assertion or an unexpected error.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

  1. Given the feature definition:

    Feature: Feature with a failed step
      Scenario: Scenario with a failed step
        Given the step throws an unexpected error
  2. And the step definition:

    [Binding]
    public static class StepDefinitions
    {
        [Given("the step throws an unexpected error")]
        public static void ThrowError()
        {
            throw new Exception("An unexpected error");
        }
    }
  3. The test result is rendered like this:

What is the expected behavior?

The test and the step should have the broken status (indicated with yellow).

What is the motivation / use case for changing the behavior?

This is a distinction that is expected by users based on their experience with other integrations.

Please tell us about your environment:

  • Allure version: 2.23.1
  • Test framework: SpecFlow@3.9.74
  • Allure adaptor: allure-specflow@2.10.0

Other information

This can be tricky to implement due to SpecFlow having multiple runners. We don't know in advance which one the user chooses. The following strategies might be considered:

  1. Try guess the assertion exceptions of the most popular runners (xUnit.net, NUnit).
  2. Provide a user with the configuration similar to how allure-nunit handles it.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions