Skip to content

set pre_filter_shard_size to 1 when includeFrozen is specified and frozen indices are queried #32742

@bczifra

Description

@bczifra

Describe the feature:
The search frozen indices documentation advises that:

While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter pre_filter_shard_size specifies a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. This filter phase can limit the number of shards significantly. For instance, if a date range filter is applied, then all indices (frozen or unfrozen) that do not contain documents within the date range can be skipped efficiently. The default value for pre_filter_shard_size is 128 but it’s recommended to set it to 1 when searching frozen indices. There is no significant overhead associated with this pre-filter phase.

Kibana has an includeFrozen setting to allow queries to search frozen indices. When this setting is enabled, the pre_filter_shard_size request property should be set to 1 when searching across frozen indices.

Describe a specific use case for the feature:
If the pre_filter_shard_size is not set to 1 then searches that include frozen indices and query against < 128 shards won't go through the filter phase. This can queries drastically slower since frozen indices that couldn't possibly match the selected date range are not skipped as part of the filter phase.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions