I don't feel necessary for test to declare supported levels.

Declared supported levels can be used for - 1) Global normalization. 2) If the test needs to supersede the module level of completeness.

I still don't follow this either. Do you have an example?

For example, if a testcase has following supported markers:

pytestmark = [
    pytest.mark.supported_completeness_level(CompletenessLevel.basic, CompletenessLevel.thorough)
]

If this testcase is executed with CLI argument --completeness_level=confident, normalization is needed to decide what level test should be run at.
In this case, the normalization logic pulls all the supported levels from the testcase during pytest_runtest_setup, and decides the normalized value should be basic.

The testcase will finally be run in basic completeness level.

I don't think this should be a marker.

By making it a maker, when a completeness level is specified. Test would be selected or deselected. This is not what we want. We want to have this as an argument so that the test could change behavior dynamically.

As per the discussion - markers seems to be an acceptable method to implement test-completeness.
Marker will not select or deselect the testcase necessarily - selecting/deselecting depends on whether pytest.skip is used.

In the proposed approach, markers enables the testcases to run at different completeness-levels. Global logic to normalize the level if the specified and defined levels do not match will be handled at the test setup level. This prevents duplication of normalization logic in every testcase. Also, after the completeness level is normalized, the testcase is free to decide what to do/execute at different levels of completeness.

Define both name and value are confusing, please stick with named levels.

Agreed, it will be much better if just named levels are used. Fixed this.

device_type(TEST_LEVEL) -> test_level(TEST_LEVEL)

Fixed now. Thanks.

This method is a little long and there's a bunch of different cases to cover. What I would suggest here is to have helper methods for the different cases, so something like:

• set_default to handle cases 1 & 2
• normalize_levels to handle case 3

And then once those two are covered you can just set the completeness level at the end and return. The comments are definitely helpful for following what is going on, but it's even easier/quicker to follow if you can just look at the method calls and figure out what each step is doing. :)

Agreed and added the methods to make it more easy to follow. Thanks.

I still don't follow this either. Do you have an example?

I'm not sure that I understand why the mark is necessary if we're just fetching it from the request and checking it in the method. How is this different than having a global normalized_level variable that we can reference or making it a field of the testbed fixture or something similar?

There might be a scope of improvement and simplification.
To explain the machinery - there are two markers being used here:

1. supported_completeness_level is used to advertise the supported levels. This marker is being used by the normalization logic to fetch the supported levels and normalize it against the specified level. This will happen during testcase setup for every testcase. I think this marker is still needed for the normalization to work during testcase setup.
2. completeness_level is basically the normalized level that a testcase will be run against. This does NOT have to be a marker. Will the global variable not cause problem if there are two or more testcases being executed simultaneously? Making a field within testbed fixture definitely sounds more reasonable than just global variable.

I didn't get that part of how this can be a field in testbed fixture. I thought normalization is done per testcase depending on what levels are supported by the testcase vs what the user requests.

You are right, Neetha. The normalization is done per testcase and testbed fixture itself will not be testcase specific. I think Danny is suggesting to add a field in the testbed fixture for every testcase during testcase setup step. It wouldn't be very different approach from just setting a global variable. I am open to suggestions.
My preference with marker was simply to stamp the testcase with its normalized level in a not-so-global variable. Hence a marker in the testcase's request data.
What are the concerns with a marker carrying this normalized value?

Update - modified the code to use just one marker instead of two. The normalized completeness level is now added to the list of supported levels.

I didn't get that part of how this can be a field in testbed fixture. I thought normalization is done per testcase depending on what levels are supported by the testcase vs what the user requests.