Currently elastic-package test provides mechanisms to test pipeline and integrated source⇒input⇒pipeline tests. These all express tests as a successful outcome, lacking an error.message value. Refining the definition of success behaviour is in #1392.

However, we do not have a way of expressing a test that intends to check the behaviour of an integration for the unhappy path. This is important because good error handling is essential for quickly identifying issues and correcting them; we are in the process of improving the reporting of API and data processing errors to allow more rapid resolution of user issues, but enhancements that we make to this aspect of integrations cannot currently be mechanically tested. Improvements to error handling and reporting require manual alteration of tests and human assessment of behaviour. This doesn't scale and makes missing regressions in behaviour inevitable.

I propose that a mechanism be added to elastic packages testing infrustructure that allows a developer to express the intention that a particular test will "fail", that is it will have some error reporting, and be able to define the shape/format of the reported error.

/cc @andrewkroh