Skip to content

Add Explicit property to [Fact] and friends #2518

@bradwilson

Description

@bradwilson

A test which is marked as explicit will normally not be run with a "run all" gesture, but will be run when asked for explicitly.

  • For the SDK, we will need some metadata to indicate explicit tests (they should have a message, much like skipped tests do today) as well as an object model object which indicates a test which was skipped because of being marked as explicit (to differentiate from tests which are skipped because of being marked as skipped). The APIs to "run all" will not run explicit tests, but any API which takes a test case list will run explicit tests if they're requested. This means that runners should be careful to segregate explicit tests when filtering (give help here when possible with the existing filter classes).
  • For theories, the theory data model in v3 has been extended to allow skipping individual data rows. The theory data model should be extended to also allow the data row to be marked as explicit.
  • For console runners, the default run all behavior will not run explicit tests (and we should add a reporting of explicit tests). A new option should be added (-explicit?) which will include explicit tests.
  • For the MSBuild runner, the default run all behavior will not run explicit tests (and we should add a reporting of explicit tests). There is no equivalent to -method or -class in the MSBuild runner. A new option should be added (Explicit=?) which will include explicit tests.
  • For the VSTest runner, the "run all" API should skip explicit tests and report explicit tests with an outcome of TestOutcome.None; the "run some" API should run any explicit test that's in the list.

I think there's some design work TBD yet. Some questions:

  • Are the only options for -explicit something like true, false, etc.? Should there, for example, be a way to run only explicit tests (f.e., -explicit only)?
  • Should -class run explicit tests in the given class? How is this impacted by -explicit?
  • Should -method run an explicit test method? How is this impacted by -explicit? What about when the method is a theory, but only some of the data has been marked as explicit?
  • Test filtering expressions should be part of the consideration for explicit tests (and probably match the features available via -explicit).

Discussion: #2483

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions