Skip to content

[8.8] Retry downsample ILM action using a new target index (#94965)#96094

Merged
elasticsearchmachine merged 1 commit intoelastic:8.8from
csoulios:backport/8.8/pr-94965
May 15, 2023
Merged

[8.8] Retry downsample ILM action using a new target index (#94965)#96094
elasticsearchmachine merged 1 commit intoelastic:8.8from
csoulios:backport/8.8/pr-94965

Conversation

@csoulios
Copy link
Copy Markdown
Contributor

Backports the following commits to 8.8:

Currently, when the ILM downample step is being retried, the same target index is used.
This can cause the subsequent downsample API invocation to index rolled up data into
shards of the target index that already exists and while the previous downsample API
invocation is still partially running (and also rolling up data into the same target shard).

Note that, the downsample step may fail in case a cluster is being restarted in a rolling 
manner (for example for an upgrade) or when the elected master node fails (the downsample 
action is coordinated from the elected master node).

This PR modfies the ILM DownsampleAction so that when DownsampleStep fails, it will retry
by going performing the following steps
  1. Cleanup existing target index, 
  2. Generate a new index name for the target index
  3. Downsample using the new target index name.

Note 1: This change may leave some garbage indices that we must find another way
to cleanup. However, the downsample process will become more resilient.

Note 2: A similar approach is used by the searchable_snapshot ILM action

Closes elastic#93580
@csoulios csoulios added :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data :StorageEngine/TSDB You know, for Metrics :Data Management/ILM+SLM DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead. >bug auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Data Management (obsolete) DO NOT USE. This team no longer exists. labels May 15, 2023
@elasticsearchmachine elasticsearchmachine merged commit d960741 into elastic:8.8 May 15, 2023
@csoulios csoulios deleted the backport/8.8/pr-94965 branch May 15, 2023 09:08
@gmarouli gmarouli added v8.8.0 and removed v8.8.1 labels May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport >bug :Data Management/ILM+SLM DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead. :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.8.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants