[Failure store] Add data stream options to the es specs by gmarouli · Pull Request #4414 · elastic/elasticsearch-specification

gmarouli · 2025-05-26T15:50:21Z

As part of the failure store feature, we introduce the data stream options. Data stream options have the following format:

{
  "failure_store": {
    "enabled": true, 
    "lifecycle": {
      "enabled": true,
      "data_retention": "7d" 
    }
  }
}

Currently they only configure the failure store and they can be set in their dedicated APIs and via the templates.

gmarouli · 2025-05-26T15:52:21Z

output/schema/validation-errors.json

+    "indices.get_data_stream_options": {
+      "request": [
+        "Request: missing json spec query parameter 'include_defaults'"
+      ],
+      "response": []
+    },


About this param. Currently, this flag has no effect. It is already in the API because initially, we considered to expose the default rollover via this flag, but we did not do this. @jbaiera can we clean it up from the API, or should I add it here and explain that it doesn't have an effect yet?

If the flag has no effect and we aren't planning on adding anything for it, we should probably just remove it from the API. Considering the api is so recently new and it hasn't been fully documented yet, I think it's a pretty low impact change to pursue.

I was hoping you would day that, I opened elastic/elasticsearch#128730

Love this, thank you!

gmarouli · 2025-05-26T15:52:55Z

...on/indices/get_data_lifecycle/examples/response/IndicesGetDataLifecycleResponseExample1.yaml

@@ -1,5 +1,5 @@
 # summary:
-description: A successful response from `GET _lifecycle/stats?human&pretty`.
+description: A successful response from `GET /_data_stream/{name}/_lifecycle?human&pretty`.


Drive-by fix, this is not the lifecycle stats API.

jbaiera

Changes LGTM!

gmarouli · 2025-06-03T11:52:09Z

specification/_json_spec/indices.get_data_stream_options.json

-      "include_defaults": {
-        "type": "boolean",
-        "description": "Return all relevant default configurations for the data stream (default: false)"
-      },


Relates to elastic/elasticsearch#128730

github-actions · 2025-06-03T12:01:31Z

Following you can find the validation results for the APIs you have changed.

API	Status	Request	Response
`cluster.allocation_explain`	🟢	4/4	4/4
`cluster.delete_component_template`	🟢	2/2	2/2
`cluster.delete_voting_config_exclusions`	🟢	1/1	1/1
`cluster.exists_component_template`	⚪	Missing test	Missing test
`cluster.get_component_template`	🟢	12/12	12/12
`cluster.get_settings`	🟢	7/7	7/7
`cluster.health`	🟢	157/157	157/157
`cluster.info`	🟢	4/4	4/4
`cluster.pending_tasks`	🟢	3/3	3/3
`cluster.post_voting_config_exclusions`	🟢	5/5	5/5
`cluster.put_component_template`	🟢	29/29	29/29
`cluster.put_settings`	🟢	59/59	58/58
`cluster.remote_info`	🟢	3/3	3/3
`cluster.reroute`	🟢	8/8	7/7
`cluster.state`	🟢	79/79	78/78
`cluster.stats`	🔴	27/27	0/27
`indices.add_block`	🟢	2/2	2/2
`indices.analyze`	🟢	246/246	246/246
`indices.cancel_migrate_reindex`	🟢	2/2	2/2
`indices.clear_cache`	🟢	4/4	4/4
`indices.clone`	🟢	5/5	5/5
`indices.close`	🟢	49/49	49/49
`indices.create_data_stream`	🟢	31/31	31/31
`indices.create_from`	🟢	4/4	4/4
`indices.create`	🔴	1179/1230	1230/1230
`indices.data_streams_stats`	🟢	4/4	4/4
`indices.delete_alias`	🔴	12/12	5/12
`indices.delete_data_lifecycle`	⚪	Missing test	Missing test
`indices.delete_data_stream_options`	⚪	Missing test	Missing test
`indices.delete_data_stream`	🟢	28/28	28/28
`indices.delete_index_template`	🟢	1/1	1/1
`indices.delete_template`	🟢	9/9	9/9
`indices.delete`	🟢	105/105	105/105
`indices.disk_usage`	🟢	6/6	6/6
`indices.downsample`	🟢	4/4	4/4
`indices.exists_alias`	🟢	37/37	37/37
`indices.exists_index_template`	⚪	Missing test	Missing test
`indices.exists_template`	🟢	15/15	15/15
`indices.exists`	🟢	39/39	39/39
`indices.explain_data_lifecycle`	⚪	Missing test	Missing test
`indices.field_usage_stats`	🟢	5/5	5/5
`indices.flush`	🟢	41/41	41/41
`indices.forcemerge`	🟢	12/12	12/12
`indices.get_alias`	🟢	68/68	68/68
`indices.get_data_lifecycle_stats`	⚪	Missing test	Missing test
`indices.get_data_lifecycle`	⚪	Missing test	Missing test
`indices.get_data_stream_options`	⚪	Missing test	Missing test
`indices.get_data_stream_settings`	🟠	Missing type	Missing type
`indices.get_data_stream`	🔴	11/11	3/11
`indices.get_field_mapping`	🟢	15/15	15/15
`indices.get_index_template`	🟢	21/21	21/21
`indices.get_mapping`	🔴	144/144	142/144
`indices.get_migrate_reindex_status`	🟢	1/1	1/1
`indices.get_settings`	🔴	76/76	47/76
`indices.get_template`	🟢	32/32	32/32
`indices.get`	🟢	55/55	55/55
`indices.migrate_reindex`	🟢	2/2	2/2
`indices.migrate_to_data_stream`	⚪	Missing test	Missing test
`indices.modify_data_stream`	⚪	Missing test	Missing test
`indices.open`	🟢	20/20	20/20
`indices.promote_data_stream`	⚪	Missing test	Missing test
`indices.put_alias`	🔴	58/58	20/58
`indices.put_data_lifecycle`	⚪	Missing test	Missing test
`indices.put_data_stream_options`	⚪	Missing test	Missing test
`indices.put_data_stream_settings`	🟠	Missing type	Missing type
`indices.put_index_template`	🟢	68/68	68/68
`indices.put_mapping`	🔴	124/128	131/131
`indices.put_settings`	🔴	56/58	58/58
`indices.put_template`	🔴	45/46	46/46
`indices.recovery`	🟢	9/9	9/9
`indices.refresh`	🟢	283/283	283/283
`indices.reload_search_analyzers`	🟢	2/2	2/2
`indices.resolve_cluster`	🟢	13/13	13/13
`indices.resolve_index`	🟢	6/6	6/6
`indices.rollover`	🔴	27/27	2/27
`indices.segments`	🔴	6/6	5/6
`indices.shard_stores`	🔴	5/5	4/5
`indices.shrink`	🟢	5/5	5/5
`indices.simulate_index_template`	🔴	6/7	5/7
`indices.simulate_template`	🔴	5/5	4/5
`indices.split`	🟢	11/11	11/11
`indices.stats`	🔴	123/123	31/122
`indices.update_aliases`	🔴	26/26	7/26
`indices.validate_query`	🟢	12/12	12/12

You can validate these APIs yourself by using the make validate target.

github-actions · 2025-06-04T06:21:00Z

The backport to 8.19 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.19 8.19
# Navigate to the new working tree
cd .worktrees/backport-8.19
# Create a new branch
git switch --create backport-4414-to-8.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 00be10ac5ace4211ff41c0c640eb9b6ecaf10f07
# Push it to GitHub
git push --set-upstream origin backport-4414-to-8.19
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.19

Then, create a pull request where the base branch is 8.19 and the compare/head branch is backport-4414-to-8.19.

pquentin · 2025-06-06T07:36:00Z

@gmarouli Thank you for this! Do you need help with the backport? I'm adding .backportrc.json in #4474 to make it easier.

(cherry picked from commit 00be10a) # Conflicts: # output/openapi/elasticsearch-openapi.json # output/openapi/elasticsearch-serverless-openapi.json # output/schema/schema.json # output/typescript/types.ts # specification/indices/_types/IndexTemplate.ts

pquentin · 2025-12-09T06:05:59Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

gmarouli · 2025-12-09T07:57:35Z

I had completely missed this....

(cherry picked from commit 00be10a) # Conflicts: # output/openapi/elasticsearch-openapi.json # output/openapi/elasticsearch-serverless-openapi.json # output/schema/schema.json # output/typescript/types.ts # specification/indices/_types/IndexTemplate.ts Co-authored-by: Mary Gouseti <mary.gouseti@elastic.co>

gmarouli added 5 commits May 26, 2025 18:30

Define new objects for data stream options and failure store

84ad9dc

Extend ComponentTemplate.ts & IndexTemplate.ts with data stream options

5f6e121

Small fix, the description was referring to the wrong URL

3d555c1

Add new data stream options requests

897ed67

Update generated files

af4e262

gmarouli requested a review from jbaiera May 26, 2025 15:50

gmarouli added specification backport 8.19 labels May 26, 2025

gmarouli changed the title ~~Add data stream options to the es spex~~ [Failure store] Add data stream options to the es spex May 26, 2025

gmarouli commented May 26, 2025

View reviewed changes

gmarouli mentioned this pull request May 26, 2025

[Index Management] Replace transport requests with js client call elastic/kibana#220614

Open

This comment was marked as off-topic.

Sign in to view

Add delete data stream options

2a88970

This comment was marked as outdated.

Sign in to view

Ensure templates can use nullable values

b8a9ab6

This comment was marked as outdated.

Sign in to view

gmarouli changed the title ~~[Failure store] Add data stream options to the es spex~~ [Failure store] Add data stream options to the es specs May 28, 2025

Merge branch 'main' into add-data-stream-options

6abf0cc

This comment was marked as outdated.

Sign in to view

jbaiera approved these changes May 30, 2025

View reviewed changes

Merge branch 'main' into add-data-stream-options

ebc82ae

This comment was marked as outdated.

Sign in to view

jbaiera mentioned this pull request Jun 2, 2025

Add documentation for failure stores. elastic/docs-content#1368

Merged

Remove from the parameter from the specification json

5477c39

gmarouli requested a review from a team as a code owner June 3, 2025 11:51

gmarouli commented Jun 3, 2025

View reviewed changes

gmarouli merged commit 00be10a into main Jun 4, 2025
8 checks passed

gmarouli deleted the add-data-stream-options branch June 4, 2025 06:19

pquentin mentioned this pull request Dec 9, 2025

[8.19] [Failure store] Add data stream options (#4414) #5807

Merged

Conversation

gmarouli commented May 26, 2025

Uh oh!

gmarouli May 26, 2025

Choose a reason for hiding this comment

Uh oh!

jbaiera May 30, 2025

Choose a reason for hiding this comment

Uh oh!

gmarouli Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

pquentin Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

gmarouli May 26, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

jbaiera left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

gmarouli Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

Uh oh!

github-actions bot commented Jun 4, 2025

Uh oh!

pquentin commented Jun 6, 2025

Uh oh!

pquentin commented Dec 9, 2025

💚 All backports created successfully

Questions ?

Uh oh!

gmarouli commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants