Skip to content

Enrich processor followup work #48988

@martijnvg

Description

@martijnvg

The issue contains improvements and new features that should be done for ingest in near to medium future.

  • Add telemetry support for the enrich policy. Interesting information to capture is the number of enrich policies by policy type, last policy execution by type and size (doc count & index size in bytes) of the enrich policy indices per type.
  • Add range enrich policy type. The range policy allows for example to match whether an ip falls into a certain range. The match field can be of type: ip_range, date_range, float_range, long_range, integer_range and double_range. Allow range types to be used for enrich matching #76110
  • Extend the get policies api to include the status of a policy and the last time a policy executed.
  • Optimize the enrich processor queue, so that when multiple search requests are executed then to combine identical search requests into a single search request in order to avoid duplicate searches in the same multi search request.
  • Introduce a LRU cache to avoid repeatable searches from the enrich processor. Add enrich node cache #76800
  • Allow enrich policy execution to be cancellable. Make enrich policy execution cancelable #77188
  • Store the result of a policy execution in the .tasks index.
  • Explore allowing multiple match fields and a combination of policy types. To allow more dynamic querying. This could perhaps be exposed as composite policy type.
  • Explore allowing enrich processor to query source index directly. In cases where flexibility is more important than performance. Introduce instant enrich policies. #73407
  • Add more surface area for stats collection on enrich indices, including average search times, total search operations, and average size of search operations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions