Problem:

It is very difficult to run our integration tests locally. This lengthens the development iteration loop, and also makes the development of new integration tests very difficult.

As a developer, when I see a pytest test, I expect to be able to run it using pytest, but that is not possible.

Solution

"best effort" testing

We should add a "best effort" mode that enables us to gracefully handle missing dependencies. It is unlikely that each local dev machine will have all the dependencies available (e.g. Java + JVM, GnuTLS), but it is very likely that each dev machine will have some of the dependencies available (S2N, OpenSSL). If I only have S2N and OpenSSL available and run pytest --best-effort I would expect that all the integration tests that have more exotic requirements like GnuTLS will be skipped, and all the other tests will be run.

removal of CI assumptions

We have baked in lots of assumptions about how the integration tests are run. Some examples.

• tests are always run using XDIST (we fail with int parsing errors otherwise)
• env variables are mixed in and substituted into pytest "config" values, which makes the tests harder to read. Additionally, some of the config values that are mandatory are not actually documented as such.

removal of unnecessary tooling

There is a lot of tooling involved in our integration tests. We should ensure that all of it is providing value. Integration tests are currently run

1. through Make
2. through CMake
3. through Nix
   I'm unsure where Nix fits into the story, but I am quite certain that we don't need both Make and CMake.

Additionally we rely on Tox, but since we only test with a single python version, I'm not sure that it's worth the extra friction

Requirements / Acceptance Criteria:

Tests can by run using pytest from a local dev instance. The requirement is not actually to switch to directly invoking pytest, but it is a good litmus test for ensuring that things are properly documented and we aren't making any CI specific assumptions.