Add `frozen_after` field to data stream lifecycle by nielsbauman · Pull Request #139042 · elastic/elasticsearch

nielsbauman · 2025-12-04T10:02:04Z

We add the new field to DataStreamLifecycle and relevant classes. We support the setting in component templates, composable templates, and the data stream lifecycle API.

Note that all of this is behind a newly added feature flag dlm_searchable_snapshots.

Resolves elastic/elasticsearch-team#2102
Resolves elastic/elasticsearch-team#2104
Resolves elastic/elasticsearch-team#2105
Resolves elastic/elasticsearch-team#2106
Resolves elastic/elasticsearch-team#2107
Resolves elastic/elasticsearch-team#2108

Copilot

Pull request overview

This PR adds support for a frozen_after field to the data stream lifecycle feature, allowing users to specify when data should be frozen. The implementation includes support across component templates, composable templates, and the data stream lifecycle API, all protected by a new feature flag dlm_searchable_snapshots.

Adds frozen_after field to DataStreamLifecycle class and related Template/Builder classes
Implements serialization/deserialization with feature flag protection
Adds REST API capability support for the new field

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamLifecycle.java	Core implementation of `frozen_after` field with parsing, serialization, and builder support
server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestPutComposableIndexTemplateAction.java	Adds capability declaration for `frozen_after` in composable index templates
server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestPutComponentTemplateAction.java	Adds capability declaration for `frozen_after` in component templates
modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/rest/RestPutDataStreamLifecycleAction.java	Adds capability declaration for `frozen_after` in data stream lifecycle API
server/src/test/java/org/elasticsearch/cluster/metadata/DataStreamLifecycleTests.java	Test coverage for `frozen_after` field mutation and random generation
server/src/test/java/org/elasticsearch/cluster/metadata/DataStreamLifecycleTemplateTests.java	Test coverage for template handling of `frozen_after` field
server/src/test/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateServiceTests.java	Test coverage for lifecycle resolution with `frozen_after` field
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.put_index_template/10_basic.yml	REST API test for index templates with `frozen_after`
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/cluster.component_template/10_basic.yml	REST API test for component templates with `frozen_after`
modules/data-streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/lifecycle/20_basic.yml	REST API test for data stream lifecycle with `frozen_after`
server/src/main/resources/transport/upper_bounds/9.3.csv	Updates transport version upper bound for the feature
server/src/main/resources/transport/definitions/referable/searchable_snapshots_dlm.csv	Defines transport version for searchable snapshots DLM feature

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

server/src/test/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateServiceTests.java

elasticsearchmachine · 2025-12-04T10:04:04Z

Pinging @elastic/es-data-management (Team:Data Management)

seanzatzdev

LGTM, thanks!

lukewhiting

One NABD comment but otherwise looks solid :-) 👍🏻

lukewhiting · 2025-12-09T09:53:28Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamLifecycle.java

-            + ", downsamplingMethod="
-            + downsamplingMethod
-            + '}';
+        return Strings.toString(this);


I believe String.toString(...) invokes the xContent serialization code which is pretty heavy weight. Any reason for this over a standard, IDE generated toString?

Any reason for this over a standard, IDE generated toString?

Yep, I have a few reasons :)

I don't like having to add every new field to this method

I don't think we invoke toString of this class often (maybe never) in production - I think this is mostly relevant in tests

We don't serialize anything big in this class. So, while XContent serialization might relatively be more expensive than simple string concatenation, I don't think the serialization of this class is necessarily heavy in absolute terms.

Let me know what you think!

dakrone

This LGTM, I left a couple of comments. I think we should validate (even internally) that the TimeValue is > 0 to avoid someone passing TimeValue.MINUS_ONE

dakrone · 2025-12-09T14:58:47Z

...main/java/org/elasticsearch/datastreams/lifecycle/rest/RestPutDataStreamLifecycleAction.java

+    private static final String SUPPORTS_FROZEN_AFTER = "dlm.frozen_after";
+    private static final Set<String> CAPABILITIES = Set.of(
+        SUPPORTS_DOWNSAMPLING_METHOD,
+        DataStreamLifecycle.DLM_SEARCHABLE_SNAPSHOTS_FEATURE_FLAG.isEnabled() ? SUPPORTS_FROZEN_AFTER : ""


Out of curiosity, is there any adverse effect to having an empty string in the set of capabilities?

I wondered the same thing. I ran the test with DataStreamLifecycle.DLM_SEARCHABLE_SNAPSHOTS_FEATURE_FLAG.isEnabled() ? SUPPORTS_FROZEN_AFTER : removed (i.e. just the empty string) and everything seemed to work just fine.

dakrone · 2025-12-09T15:00:02Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamLifecycle.java

    public static final TransportVersion ADDED_ENABLED_FLAG_VERSION = TransportVersions.V_8_10_X;
+    private static final TransportVersion INTRODUCE_LIFECYCLE_TEMPLATE = TransportVersion.fromName("introduce_lifecycle_template");
    public static final TransportVersion ADD_SAMPLE_METHOD_DOWNSAMPLE_DLM = TransportVersion.fromName("add_sample_method_downsample_dlm");
+    public static final TransportVersion SEARCHABLE_SNAPSHOTS_DLM_TV = TransportVersion.fromName("searchable_snapshots_dlm");


We should probably make these all either public, or all private, to match each other, right?

I don't have strong feelings. I don't see much value in making the private ones public if there's no need for them to be public, but I'm empathetic to consistency :)

dakrone · 2025-12-09T15:01:33Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamLifecycle.java

+            if (value == null) {
+                return null;
+            } else {
+                return TimeValue.parseTimeValue(value, FROZEN_AFTER_FIELD.getPreferredName());


We should probably validate that the value is > 0, right?

Ah yeah, good one! I'll add the check to the constructor, that seems like the best place to add the validation.

I remember thinking about whether we should also validate that frozen_after is before retention and after any downsampling (but I forgot to comment that). Can you think of a reason not to add that validation too?

dakrone · 2025-12-09T15:04:26Z

...rc/main/java/org/elasticsearch/rest/action/admin/indices/RestPutComponentTemplateAction.java

    public static final String SUPPORTS_FAILURE_STORE = "data_stream_options.failure_store";
    private static final String COMPONENT_TEMPLATE_TRACKING_INFO = "component_template_tracking_info";
    static final String SUPPORTS_DOWNSAMPLING_METHOD = "dlm.downsampling_method";
+    static final String SUPPORTS_FROZEN_AFTER = "dlm.frozen_after";


Can we re-use the other one from PutDataStreamLifecycle? Perhaps we can put the string into a common place and reference them from each different module?

I was able to use the one from RestPutComponentTemplateAction in RestPutDataStreamLifecycleAction - I didn't push yet.

* upstream/main: Update text_similarity_rank_retriever to support inference ID as an argument when reranking on chunks (elastic#137397) Mute org.elasticsearch.test.rest.yaml.CssSearchYamlTestSuiteIT test {p0=search.retrievers/result-diversification/10_mmr_result_diversification_retriever/Test MMR result diversification multiple indexes} elastic#139826 Mute org.elasticsearch.index.mapper.SkipperSettingsTests testTSDBSkipperSettingDefaults elastic#139824 Unmute test fix elastic#129517 (elastic#139782) Add back support for deserializing old refresh token in test (elastic#139811) Add documentation for exponential_histogram field type (elastic#139684) make ES|QL sample CSV test looser (elastic#139814) Add `frozen_after` field to data stream lifecycle (elastic#139042) Quieten many `ERROR` logs to `WARN` (elastic#139799)

Add frozen_after field to data stream lifecycle

0a0b137

nielsbauman requested a review from Copilot December 4, 2025 10:02

elasticsearchmachine added the v9.3.0 label Dec 4, 2025

Copilot AI reviewed Dec 4, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateServiceTests.java Show resolved Hide resolved

nielsbauman added >non-issue :StorageEngine/Data streams Data streams and their lifecycles labels Dec 4, 2025

nielsbauman marked this pull request as ready for review December 4, 2025 10:03

elasticsearchmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Dec 4, 2025

seanzatzdev approved these changes Dec 8, 2025

View reviewed changes

lukewhiting approved these changes Dec 9, 2025

View reviewed changes

dakrone approved these changes Dec 9, 2025

View reviewed changes

Reuse capability string

c3626c1

nielsbauman added the test-release Trigger CI checks against release build label Dec 9, 2025

elasticsearchmachine added v9.4.0 and removed v9.3.0 labels Dec 17, 2025

nielsbauman added 3 commits December 18, 2025 10:22

Add > 0 validation

eb6ed5d

Merge branch 'main' into add-frozen-after-field

4e0df7a

Add tests

dbfd088

nielsbauman enabled auto-merge (squash) December 18, 2025 09:44

nielsbauman added 4 commits December 18, 2025 12:07

Fix non-snapshot tests

ec26a20

Merge branch 'main' into add-frozen-after-field

cefe5ae

Merge branch 'main' into add-frozen-after-field

3e80c45

Merge branch 'main' into add-frozen-after-field

c6aeb6d

nielsbauman disabled auto-merge December 19, 2025 14:05

nielsbauman merged commit 54b35d6 into elastic:main Dec 19, 2025
34 of 37 checks passed

nielsbauman deleted the add-frozen-after-field branch December 19, 2025 14:05

Conversation

nielsbauman commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 4, 2025

Uh oh!

seanzatzdev left a comment

Choose a reason for hiding this comment

Uh oh!

lukewhiting left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nielsbauman commented Dec 4, 2025 •

edited

Loading