Steps to reproduce:

• Some skip-if conditions are too generic and include only the os (eg: skip-if = "os == 'linux'")
• Some skip-if conditions are too specific and become difficult to understand (eg: "os == 'linux' && os_version == '18.04' && processor == 'x86_64' && debug && socketprocess_networking")

Actual results:

The skipfails script retrieves a lot of information from failed tasks and creates very complex conditions to skip tests. These conditions can become very difficult to read and understand.

A lot of skip-if conditions were also entered manually and only include the os, which is too generic and may skip valid tests on more recent versions of the OS.

Expected results:

The skipfails script should only use the os, the os_version and the processor type (the OOP combo) for generating skip-if conditions. These conditions should be a lot easier to understand and make updating platforms less of a headache.

The script should also detect conditions too generic or too specific and rewrite them in OOP format.

I would add in build config (like opt, debug, asan, tsan, ccov), at this point we end up with something not as wide spread as before, but something closer to specific details. Unfortunately in many cases ignoring variants would result in up to 4 versions of the tests disabling.

Including the build type adds a lot of lines when a test has to be skipped for a whole platform. Here is an example of a test which has to be skipped on all Linux variants:

  "os == 'linux' && os_version == '18.04' && processor == 'x86_64' && asan", # Bug TBD (secondary)
  "os == 'linux' && os_version == '18.04' && processor == 'x86_64' && debug", # Bug TBD (secondary)
  "os == 'linux' && os_version == '18.04' && processor == 'x86_64' && opt", # Bug TBD (secondary)
  "os == 'linux' && os_version == '18.04' && processor == 'x86_64' && tsan", # Bug TBD (secondary)
  "os == 'linux' && os_version == '22.04' && processor == 'x86_64' && debug", # Bug TBD (secondary)
  "os == 'linux' && os_version == '22.04' && processor == 'x86_64' && opt", # Bug TBD (secondary)
  "os == 'mac' && os_version == '10.15' && processor == 'x86_64' && debug", # Bug TBD (secondary)
  "os == 'mac' && os_version == '10.15' && processor == 'x86_64' && opt", # Bug TBD (secondary)

Would this be acceptable?

in general that is what I am going for- we are getting a lot of pushback on this- so I think there are 2 optimizations to make:

1. if all test variants are failing, do not include test variants (an edge case is a new test variant would then be automatically skipped).
2. if all build_configs are failing, do not include build_config.

These are both optimizations for the human eye- and both have a problem of needing a complete list (on that day) of all build_config and test_variants we support. Likewise in the future as we automate more test manifest conditions, this will inevitably become a longer list, so the optimizations are a temporary fix to make developers feel better about the readability.

If we could generate a list of all permutations of os/os_version/build_config/test_variant for a given test suite, then theoretically we could create an optimization within skipfails.py that would:

1. if ALL test_variants are listed on a skip-if, then remove all test_variants and remove duplicate lines
2. if there are no test_variants and all build_configs are listed on a skip-if, then remove all build_configs and remove duplicate lines

We could generate a list of all permutations by parsing a taskgraph artifact.

:aryx, does this sound good? Arnaud, any further thoughts?

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #12)

2. if all build_configs are failing, do not include build_config.

build configs: debug, opt, asan, tsan, ccov, ...

1. if ALL test_variants are listed on a skip-if, then remove all test_variants and remove duplicate lines

test variants: configured by a preference or environment variable change in the test task, e.g. http/3, new socketprocess interface.

Comment 12 sounds reasonable.

another issue with collecting the list of build_types/test_variants is that on new OS we don't have data in the taskgraph. An example would be adding tests to macosx 14.70 where we don't have any scheduled right now. In this case we would be running the tests on a new machine pool with custom taskgraph commands. This scenario would need to be considered with a solution as well.

Retrieving the list of all build types and test variants may be tricky, I will look into this.

Tries to simplify the conditions as much as possible by checking the
possible permutations.

https://hg.mozilla.org/mozilla-central/rev/2182033571d6
https://hg.mozilla.org/mozilla-central/rev/912a6b41411e
https://hg.mozilla.org/mozilla-central/rev/39bdb6eba77d
https://hg.mozilla.org/mozilla-central/rev/906e9fe9c185
https://hg.mozilla.org/mozilla-central/rev/99fec8dd463d
https://hg.mozilla.org/mozilla-central/rev/a2e3c1ee92be

https://hg.mozilla.org/mozilla-central/rev/bfb26b488b87

https://hg.mozilla.org/mozilla-central/rev/f5c826f8d3eb

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

https://hg.mozilla.org/mozilla-central/rev/9b16e64a8de3

https://hg.mozilla.org/mozilla-central/rev/d982fe6e55c8
https://hg.mozilla.org/mozilla-central/rev/5a22d97c9927