-
Notifications
You must be signed in to change notification settings - Fork 25.8k
Add an ILM action to rollup an index #48003
Copy link
Copy link
Closed
Labels
:Data Management/ILM+SLMDO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead.DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead.:StorageEngine/RollupTurn fine-grained time-based data into coarser-grained dataTurn fine-grained time-based data into coarser-grained data>featureTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)Team:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.
Description
Rollup should no longer be a continuous running job. Instead, it should be an action that can be triggered on ILM-managed indices. The action should iterate over all grouping tuples (dimensions) and calculate aggregate metrics (min/max/avg/value_count) generating one document per grouping tuple (dimension).
In the context of this issue we should implement the following:
- Implement a "grouping tuple" field type to generate the tuple at index time for the document.
- Implement a "rollup metric" field type to store aggregation results (min/max/count/avg etc). This metric should provide the correct information to requesting aggregator (e.g.
avgagg onrollup_metricwill internally fetch sum + count) - Store document count somewhere on the document (docvalue field?)
- Create an ILM rollup action
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
:Data Management/ILM+SLMDO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead.DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead.:StorageEngine/RollupTurn fine-grained time-based data into coarser-grained dataTurn fine-grained time-based data into coarser-grained data>featureTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)Team:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.
Type
Fields
Give feedbackNo fields configured for issues without a type.