Skip to content

improve pipeline.ordered setting to rebatch before outputs #11550

@colinsurprenant

Description

@colinsurprenant

In #11524 we introduced the pipeline.ordered setting to solve the single-worker event ordering preservation that was broken under the Java execution.

This fix is suboptimal in the sense that the implementation drips a batch event down the filter+output computation leading to having the outputs multi_receive work with events one-by-one instead of a complete batch. This will negatively impact performance for outputs such as the elasticsearch output which leverages a batch to perform a bulk operation.

As discussed in #11524 (comment) one potential fix would be to try to re-aggregate a batch after the filtering but before the outputs.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions