Skip to content

ILM introduce the check-ts-end-time-passed step#100179

Merged
elasticsearchmachine merged 5 commits intoelastic:mainfrom
andreidan:ilm-wait-for-end-time
Oct 3, 2023
Merged

ILM introduce the check-ts-end-time-passed step#100179
elasticsearchmachine merged 5 commits intoelastic:mainfrom
andreidan:ilm-wait-for-end-time

Conversation

@andreidan
Copy link
Copy Markdown
Contributor

This introduces a new ILM step, the check-ts-end-time-passed step that'll wait for the index.time_series.end_time to lapse for TS indices before allowing ILM to proceed with the execution of the following actions:

  • downsample
  • forcemerge
  • readonly
  • searchable_snapshot
  • shrink

TSDS indices are allowed to receive writes until the configured index.time_series.end_time time passes. This makes sure ILM doesn't block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696

This introduce a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions:
* downsample
* forcemerge
* readonly
* searchable_snapshot
* shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.
@andreidan andreidan added >bug :Data Management/ILM+SLM DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead. v8.11.0 v8.10.3 labels Oct 3, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Oct 3, 2023
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @andreidan, I've created a changelog YAML for you.

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@andreidan
Copy link
Copy Markdown
Contributor Author

@elasticmachine run elasticsearch-ci/part-1

@andreidan
Copy link
Copy Markdown
Contributor Author

andreidan commented Oct 3, 2023

@elasticmachine run elasticsearch-ci/part-2

Failure tracked already in #96723

Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one testing comment. LGTM otherwise.

}
Instant configuredEndTime = IndexSettings.TIME_SERIES_END_TIME.get(indexMetadata.getSettings());
assert configuredEndTime != null : "a time series index must have an end time configured but [" + index.getName() + "] does not";
if (Instant.now().isBefore(configuredEndTime)) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe provide current time as a lambda function? This would allow unit tests to not rely on current time.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! Great shout. Done in a6ffc7d

Thanks Martijn

@andreidan
Copy link
Copy Markdown
Contributor Author

@elasticmachine run elasticsearch-ci/part-3

@andreidan andreidan added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 3, 2023
@elasticsearchmachine elasticsearchmachine merged commit 8bf8fc1 into elastic:main Oct 3, 2023
@andreidan andreidan deleted the ilm-wait-for-end-time branch October 3, 2023 13:43
andreidan added a commit to andreidan/elasticsearch that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes elastic#99696

(cherry picked from commit 8bf8fc1)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
andreidan added a commit to andreidan/elasticsearch that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes elastic#99696
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

💚 Backport successful

Status Branch Result
8.10

elasticsearchmachine pushed a commit that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696
andreidan added a commit that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696

(cherry picked from commit 8bf8fc1)

Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) >bug :Data Management/ILM+SLM DO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead. Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.10.3 v8.11.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TSDS data streams can route writes to downsampled indices after rollover

3 participants