There are 2 places where we use arbitrary decisions on times:
Sleep in the workflow, since the backing store is ready but the free storage in the test pool is 0 / the mode is INITIALIZING. It is only a bypass until we resolve this issue.
Setting timeout in the job waiting for it to complete - the job will always complete - we set a long timeout.

It will run on master in cases of the pull request and push.
We thought about a nightly run - it creates questions regarding the responsibility to check that this check passed and if not to fix the merged code... and we decide to avoid it. In order to collect observations without interrupting the team's work, this action would run manually and scheduled (twice during the night).

@shirady on the test list file, we should add a comment on each test to describe why we are ignoring/black listing it

We want the test list to be separated from the configuration and from the files that are running them - we used the flag with the paths to the lists.
We thought about two options format for the tested files: JSON and text file.
In JSON we cannot write comments as we do in code, since it creates an error after parsing it.
The solution is to add the comment of the reason as a key-value pair. for example:

{
   "_comment": "a faulty test on ceph side",
   "test": "s3tests_boto3.functional.test_s3.test_bucket_create_exists" 
}

we want a simple way to add and remove tests (for example if someone wants to run them in a different namespace), and using the JSON this way was not a simple way to add/remove tests.
We use a simple text with the list of tests as they were saved in the code before and as they are printed in the logs - it is super easy to add/remove tests.

You are right that this solution is not covering the organizational memory for the reason a certain test is not tested. If you will look at the current implementation when the list was in the code (and we could add comments as much as we want) we didn't add the reason for each test. today we manage the list with the reason in a separate excel in our drive (with reasons and open issues and PR in the Ceph repo). Of course, we can approve it, but I'm not sure I'll do it in the scope of this PR.

Update:

Currently, the VM we use does not have sufficient resources to run all the tests.
This causes glitches: we can see that there is a certain moment when a couple of tests are failing (kind of a “block”), this moment is changing, and the failed test list is not the same.
Anyway, on my machine the passing list is consistent.
Note: we now use one worker, so it is not related to a concurrency problem.

What we did so far:

instead of using nb install with flag mini we tried to moc this command and run the resources we need.
instead of:
./build/_output/bin/noobaa-operator install --mini --noobaa-image='noobaa-core:s3-tests'
we tried:

nb crd create
nb operator install

First Try:
nb system create --db-resources='{ "limits": {"cpu": "800m","memory": "1G"}, "requests": {"cpu": "800m","memory": "1G"}}' --core-resources='{ "limits": {"cpu": "300m","memory": "1G"}, "requests": {"cpu": "300m","memory": "1G"}}' --endpoint-resources='{ "limits": {"cpu": "500m","memory": "1G"}, "requests": {"cpu": "500m","memory": "1G"}}' --noobaa-image='noobaa-core:s3-tests'

But we got: Insufficient cpu for the db pod

Second Try:
./build/_output/bin/noobaa-operator system create --db-resources='{ "limits": {"cpu": "500m","memory": "1G"}, "requests": {"cpu": "500m","memory": "1G"}}' --core-resources='{ "limits": {"cpu": "300m","memory": "1G"}, "requests": {"cpu": "300m","memory": "1G"}}' --endpoint-resources='{ "limits": {"cpu": "400m","memory": "1G"}, "requests": {"cpu": "400m","memory": "1G"}}' --noobaa-image='noobaa-core:s3-tests'

But we got: Insufficient cpu for the endpoint pod

From what I saw in the new observations: when this workflow runs in parallel to itself (manually run it twice) it creates glitches. When it runs alone it not fails the tests.
I even added a scheduled run (same as here) - one failed and one pass. So, we probably will need to find the right time to run it inside the repo (when only this job will be running).