Skip to content

mark flaky / broken tests in a way to allow skipping them #900

@dirk-thomas

Description

@dirk-thomas

pytest already defines a marker for exactly that purpose: pytest.mark.xfail. I would suggest to use that for Python tests rather than coming up with a custom name. In CTest we can use a label with the same name (xfail).

The triggered CI / nightly jobs should be updated to skip tests with that marker / label. It probably makes sense to have a set of (nightly) jobs which only run those tests to keep track of the number of tests marker as such (as well as their status).

Beyond this technical approach there needs to be a "rule / policy" that tests marked like this (and therefore effectively being disabled) to spend resources on these tests to be able to reenable them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions