Skip to content

[backport] Adds a new Rollup Action (#64900)#65393

Merged
talevy merged 1 commit intoelastic:7.xfrom
talevy:7x-64900
Nov 24, 2020
Merged

[backport] Adds a new Rollup Action (#64900)#65393
talevy merged 1 commit intoelastic:7.xfrom
talevy:7x-64900

Conversation

@talevy
Copy link
Copy Markdown
Contributor

@talevy talevy commented Nov 23, 2020

This commit adds a new endpoint for Rollup V2, a new way to rollup indices.

Instead of relying on a cron-job, this action will rollup a whole index on the spot.

When an index is rolled up using a Rollup Config, it does the following

  1. check that original index is read-only
  2. runs an aggregation and indexes results into a temporary hidden rollup index
  3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling)
  4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds
    custom index-metadata with data about what the rollup index's original index is so that its
    group information in RollupMetadata can be looked up

example usage:

POST /_rollup_vtwo/index
{
    "rollup_index": "index_rolled",
    "groups": {
        "date_histogram": {
            "field": "date",
            "calendar_interval": "1M"
        },
        "terms": {
            "fields": ["unit"]
        }
    },
    "metrics": [
        {
            "field": "temperature",
            "metrics": ["sum"]
        }
    ]
}

relates #42720.

@talevy talevy added :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data backport Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Nov 23, 2020
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

This commit adds a new endpoint for Rollup V2, a new way to rollup indices.

Instead of relying on a cron-job, this action will rollup a whole index on the spot.

When an index is rolled up using a Rollup Config, it does the following

1. check that original index is read-only
2. runs an aggregation and indexes results into a temporary hidden rollup index
3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling)
4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds
    custom index-metadata with data about what the rollup index's original index is so that its
    group information in RollupMetadata can be looked up

example usage:

```
POST /_rollup_vtwo/index
{
    "rollup_index": "index_rolled",
    "groups": {
        "date_histogram": {
            "field": "date",
            "calendar_interval": "1M"
        },
        "terms": {
            "fields": ["unit"]
        }
    },
    "metrics": [
        {
            "field": "temperature",
            "metrics": ["sum"]
        }
    ]
}
```

relates elastic#42720.
@talevy talevy changed the title Adds a new Rollup Action (#64900) [backport] Adds a new Rollup Action (#64900) Nov 23, 2020
@talevy
Copy link
Copy Markdown
Contributor Author

talevy commented Nov 23, 2020

run elasticsearch-ci/packaging-sample-windows

@talevy talevy merged commit da77dcd into elastic:7.x Nov 24, 2020
@talevy talevy deleted the 7x-64900 branch November 24, 2020 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants