What it comes to testing right now we have this:

• Unit tests that test all processors in isolation.
• Java integration tests, that test all the Java level api integration.
• Rest tests running in a real ES node with ingest installed, that test the rest api level integrations and test if processors are able to be used in pipelines. (e.g. not running in class loader issues or security manager issues)

What currently is lacking is that there aren't really tests, that test pipelines with all the possible combination of processors that can exist. Adding a true random test for this is a lot of work and we will end up with a test that will be hard to maintain, but what we can do it add certain pipeline configurations that are likely to be common and test that with realistic documents. This test can just extend from ESTestCase and we can create the pipeline configurations programatically. (no need to start a test with ES nodes)