Elasticsearch returns status code 413 when a bulk request exceeds the size limit. A user can either increase the http.max_content_length in ES or decrease the bulk_max_size in the Beat to overcome such failures.
However, when this error happens and the beat is using a GuaranteedSend publisher method the current implementation can lead to an infinite retry, sending the same request to ES.
This might result in not being able to ingest any more events.
It might be worth exploring to use a special handling for the batch when the request size exceeds a limit, e.g. split it in half.
Elasticsearch returns status code
413when a bulk request exceeds the size limit. A user can either increase thehttp.max_content_lengthin ES or decrease thebulk_max_sizein the Beat to overcome such failures.However, when this error happens and the beat is using a
GuaranteedSendpublisher method the current implementation can lead to an infinite retry, sending the same request to ES.This might result in not being able to ingest any more events.
It might be worth exploring to use a special handling for the batch when the request size exceeds a limit, e.g. split it in half.