[Ingest] Ingest better bulk failure handling#14888
[Ingest] Ingest better bulk failure handling#14888martijnvg merged 1 commit intoelastic:feature/ingestfrom
Conversation
0d4c46a to
6f40576
Compare
There was a problem hiding this comment.
can we call it markCurrentItemAsFailed ? Also can we not retrieve from the bulkrequest given currentSlot the index request rather than requiring it as an argument?
|
change looks great @mvg thanks a lot. I left some comments around code style and design plus tests, let me know what you think. |
|
@javanna I'v updated the PR. |
There was a problem hiding this comment.
I see that we are now doing new Object[]{pipelineId}. I don't get why though, wasn't just pipelineId ok?
There was a problem hiding this comment.
at runtime the wrong method was used... and there for the variable placeholder didn't work and the variable was appended at the end of the log line
By using object[] the right format method gets used.
There was a problem hiding this comment.
I'll remove the LoggerMessageFormat usage, lets go back the plain old string concatenation.
|
I left a few more comments, looks good, just a few small things that we can improve left I think. let me know what you think. |
1c90d20 to
8964ebd
Compare
|
@javanna thanks, I updated the PR. |
There was a problem hiding this comment.
with the current randomizations, do we ever test this branch? If not would it be worth to write a specific test for it?
|
much better thanks @martijnvg . I left a question around testing, LGTM otherwise |
…y fail a bulk item.
8964ebd to
8b1f117
Compare
With this change if ingest pipeline fails only the index request in the bulk fails whereas right now the entire bulk request is failed.
If pipeline processing fails then the index request gets reported as a failure in the bulk response in its respective place. The index request will not be executed after pipeline processing has been completed.
If pipeline failures occur a new bulk request will be created based on the current bulk request. Only index requests that have successfully been processed by the pipeline will be included in this new bulk request. Then when sending back the bulk response the new bulk request will map its response items in such a way that the order of all items and errors is the same and matches with the original bulk request that was sent.