How are we deciding between before and beforeEach in these tests? Unless there's a huge performance cost my preference is always for beforeEach to minimize shared state between tests.

I understand your point, but there are specific scenarios were there is not going to be shared state of the tests that can impact the behaviour and is going to save us execution time.

What are the scenarios you're referring to, here? The use of cleanKibana in before only works if each test is allowed to complete its teardown:

• If a test fails after it's created a rule and deleteRule is part of the test itself, then the rule SO will be persisted to the next test
• If a test fails before it's created a rule and deleteRule is part of an after* hook, deleteRule will fail and cause cypress to skip the remaining tests

I personally would rather have a slow test than a flaky one. Performance and reliability are (with the exception of timeouts) mostly independent issues, and reliability is the focus at the moment. IMO we've also got many more options for improving performance/speed of these tests external to the tests themselves (more CI groups, beefier CI machines, etc.) such that I don't think we need to be sacrificing reliability for performance right now.

If all our tests use cleanKibana in a beforeEach, I don't think we need the same in afterEach.

However, that does bring up a general question about teardown: what is the cypress suite's teardown responsibility as a whole? Is cypress relegated to its own CI group and so we don't have to worry about cleaning up for the "next" suite? Is there a possibility that tests will be added to the cypress group and we do need to start caring about this?

I'm thinking about the situations that have bit us in the past, namely incomplete teardown and shared state, which are strongly related. We should do whatever we can to minimize those and isolate tests.

Why was this removed? Can we remove the package as well? It seems unused now.

Because we are not using it. I didn't remove the package since now we are using a shared package.json and we are not the only ones using cypress.

This means that we ignore any uncaught applications, right? This seems like a big change unless I'm misunderstanding. Were we seeing false negatives?

Without that line, cypress fails each time that an error is raised in the console log. Do we want this behaviour to be caught? Because right now there are some tests failing because of that although the UI looks fine.

Got it. It would be nice to log the error somewhere rather than discard it, but that might be too noisy for CI logs 🤷‍♂️ ? I defer to you.

I think these should go back into the afterEach. I had mentioned offline that, due to cypress skipping the suite if the test fails, we should move any test-specific teardown (e.g. deleteRule) into the test itself to avoid unnecessary skips, but I don't think that these unloads fall in that category.

This is a bit of a tangent, but I just had a couple thoughts:

• Keeping test setup/teardown symmetrical is a PITA
  • reorienting this to do teardown, then setup all within setup would reduce this complexity
    • This would address/sidestep the issue of test failures, which can currently cause teardown not to execute
    • We could lessen developer burden by defining this as a general function(s)

// in test file
describe('my test', () => {
  detectionEngineHooks();
  
  it('does a thing', () => {});
});

// in helper file
const detectionEngineHooks = () => {
  beforeEach(() => {
    cleanKibana(); // deletes `.kibana*`, loads `empty_kibana`
    cleanSignals(); // deletes `.siem-signals*`, hits detections endpoint to create index
    cleanLoad('alerts'); // see below
  });
};

const cleanLoad = (archive) => {
    unload(archive); // might need to try/catch this?
    load(archive);
}