In #11524 we introduced the pipeline.ordered setting to solve the single-worker event ordering preservation that was broken under the Java execution.
This fix is suboptimal in the sense that the implementation drips a batch event down the filter+output computation leading to having the outputs multi_receive work with events one-by-one instead of a complete batch. This will negatively impact performance for outputs such as the elasticsearch output which leverages a batch to perform a bulk operation.
As discussed in #11524 (comment) one potential fix would be to try to re-aggregate a batch after the filtering but before the outputs.
In #11524 we introduced the
pipeline.orderedsetting to solve the single-worker event ordering preservation that was broken under the Java execution.This fix is suboptimal in the sense that the implementation drips a batch event down the filter+output computation leading to having the outputs
multi_receivework with events one-by-one instead of a complete batch. This will negatively impact performance for outputs such as the elasticsearch output which leverages a batch to perform a bulk operation.As discussed in #11524 (comment) one potential fix would be to try to re-aggregate a batch after the filtering but before the outputs.