Something like this:

threadPool.executor(ThreadPool.Names.MANAGEMENT).execute(() -> {
    payload.pipeline().execute(data);
    responses.add(new SimulatedItemResponse(data));         
});

cool.

then I think all of this can be moved into the SimulatePipelineRequestPayload class (might need to rename class to be shorter 😄)

and called like

payload.execute(threadPool)

if we go for supporting get too, use the util RestActions#hasBodyContent and getRestContent instead, those support the body passed in as the source query string parameter, which all of the core apis should support when they support a get with body (for clients that don't support sending a body with get)

Maybe move the pipeline factory to be a field of the factory class?

I believe this needs to be updated to include the metadata as it was given in the input

so something more like

Map<String, Object> dataDoc = ...
dataDoc.put("_index", data.getIndex());
dataDoc.put("_type", data.getType());
dataDoc.put("_id", data.getId());
builder.field(Fields.DOCUMENT, dataDoc);

Although, I think it would be better for Data to provide a getDocumentWithMetadata to do this sort of thing so it can be re-used and agnostic to any new fields we have in metadata.

what do you think @javanna, @martijnvg ?

unfortunately Data can't implement ToXContent... maybe we should just have a static helper method that serialises the Data with an XContentBuilder?

oh I just realized that I suggested below to make Data implement both Streamable and ToXContent...which is not correct as it would mess up dependencies. I think sooner or later we should extract these classes to a different module to prevent mistakes and confusion and enforce the no deps rule that we follow. We should have some shared code around outputting Data to ToXContent and serializing it too. Shall we have an enhanced version of Data as part of the plugin that accepts a Data instance but also implements ToXContent and Streamable?

I don't see yet specific equals/hashcode tests, I think we could benefit from those. I know that they have been generated and are most likely fine at the moment, but pretty sure without tests they will get out of sync with future modifications.

added a couple,

found a really annoying problem that needs to be worked around...

Throwable comparison does not work...

Objects.equals(Throwable, Throwable)

I added logic to say that two Throwables are equivalent so long as they are of the same class... I am not a fan of this solution, have any ideas?

This needs cleaning up.

testing whether two Throwable objects are equal is not working on the object level, so I decided to just test whether they are of the same class type

I didn't realize we would return NPE. Is that because a field is not there? In general I think we should return nicer errors as much as we can.

I believe this NPE is thrown by the MutateProcessor.

Data#getProperty should wrap the exception is the suggestion?

yea ok let's do it on a separate issue though, doesn't need to be part of the PR, although it will require changing these tests ;)