Synthetic id field for time_series indices (remove feature flag) by burqen · Pull Request #144184 · elastic/elasticsearch

burqen · 2026-03-13T11:24:42Z

This commit removes the feature flag for synthetic id and thereby makes
it officially available in production. The setting is controlled by
index.mapping.synthetic_id which default to true for new time_series
indices, as long as they use the default codec.

Indices with synthetic _id fields doesn't store the id physically on
disk. Instead, the id is materialized synthetically from other document
fields. This reduces the footprint on disk for time_series indices.

NOTE! The changes themselves are not super interesting, but this will enable synthetic id everywhere, turned ON by default, and as such might need some extra care.

The slightly more interesting changes are in:

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java
server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
server/src/main/java/org/elasticsearch/index/IndexSettings.java

This commit removes the feature flag for synthetic id and thereby makes it officially available in production. The setting is controlled by `index.mapping.synthetic_id` which default to `true` for new time_series indices, as long as they use the default codec. Indices with synthetic _id fields doesn't store the id physically on disk. Instead, the id is materialized synthetically from other document fields. This reduces the footprint on disk for time_series indices.

elasticsearchmachine · 2026-03-19T13:16:20Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2026-03-19T13:16:32Z

Hi @burqen, I've created a changelog YAML for you.

burqen · 2026-03-19T15:26:21Z

Test failures:

ServerlessClientYamlTestSuiteIT.test {yaml=tsdb/150_runtime_fields/can't shadow metrics}, there is a similar issue on can't shadow dimensions here: https://github.com/elastic/elasticsearch-serverless/issues/5888
LogsdbTestSuiteIT, know issue [CI] LogsdbTestSuiteIT 20_ignored_source yaml tests failing in release-tests #144540
GenerativeIT , is known to be flaky

Triggering another run

…e-feature-flag

Add logging for index settings and node versions after index creation in ClientYamlTestClient

…etic-id-default-and-remove-feature-flag

burqen · 2026-03-20T14:11:39Z

Analysis of id mismatch failure in MixedClusterClientYamlTestSuiteIT

Summary: Both old node dies because of known issue, #143980. With only new nodes left, time_series index use synthetic id by default which is counter to the test expectation. I don't see a way to protect against this and i think it's reasonable to assume that the test cluster stays intact. If it falls apart, it's because of another bug somewhere.

Full analysis

Relevant test failure

REPRODUCE WITH: ./gradlew ":qa:mixed-cluster:v9.3.3#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=tsdb/25_id_generation/routing_path matches object}" -Dtests.seed=1EC87079E8768BBC -Dtests.bwc=true -Dtests.locale=az-Cyrl-AZ -Dtests.timezone=Asia/Chita -Druntime.java=25

MixedClusterClientYamlTestSuiteIT > test > test {p0=tsdb/25_id_generation/routing_path matches object} FAILED
    java.lang.AssertionError: java.lang.AssertionError: 
    Expected: "8bgiqW9JKwAyp1bZAAABeRnRGTM"
         but: was "JFvylKRG_nnAwbD-SKLruAAWrvVxXB-xBpIYlWsMx6RFR8fHaX___obmLubMqSK48Q"

This is a mixed cluster test that has two 9.3.3-SNAPSHOT nodes, and two 9.4.0-SNAPSHOT nodes.

For some reason the index uses synthetic id by default (it is not set in tsdb/25_id). The only way this can happen (I think) is if a 9.4.0 node is master and it uses a late index version (not supported by 9.3.3). This should not be possible in a mixed cluster and it would be a pretty serious bug.

My current theory is that the two older nodes have died and the cluster is operating on only the two newer nodes. They would then give IndexVersion.current() to new indices which would explain the default behavior. This is somewhat supported by lots of test failing with java.io.UncheckedIOException: java.net.ConnectException: Connection refused.

4bc666f add temporary logging to try and catch index version and the state of cluster nodes when creating the index used by tsdb/25_id. (TO BE REVERTED BEFORE MERGE)

EDIT: The test uses synthetic id because both old nodes disconnect from the cluster (leaving only new nodes left which will default to synthetic id for time_series indices)

Cluster Timeline — `id_generation_test` Analysis

v9.3.3-0 and v9.3.3-1 are the "upgraded" nodes — they appear running both 9.3.3 (before upgrade) and 9.4.0 (after restart). v9.3.3-2 and v9.3.3-3 never upgrade.

Setup sequence

Time	Event
15:52:04	Cluster forms. v9.3.3-1 elected master. All 4 nodes at 9.3.3
15:52:06	v9.3.3-0 disconnects (upgrading)
15:52:33	v9.3.3-0 rejoins at 9.4.0, wins new election (term 3), becomes master
15:52:34	v9.3.3-1 disconnects (upgrading)
15:53:01	v9.3.3-1 rejoins at 9.4.0

Index creation cluster membership

#	Time	Nodes in cluster	Notes
1	15:53:18	v9.3.3-0 (9.4.0, master), v9.3.3-1 (9.4.0), v9.3.3-2 (9.3.3), v9.3.3-3 (9.3.3)	Full 4-node mixed cluster
—	15:54:16		v9.3.3-2 disconnects (reason: disconnected)
2–6	15:54:48–15:59:01	v9.3.3-0 (9.4.0, master), v9.3.3-1 (9.4.0), v9.3.3-3 (9.3.3)	3-node mixed cluster
—	16:01:24		v9.3.3-3 disconnects (reason: disconnected)
7–15	16:16:09–16:21:41	v9.3.3-0 (9.4.0, master), v9.3.3-1 (9.4.0)	2-node fully-upgraded cluster

Key observations

Master is always v9.3.3-0 at 9.4.0 — it wins the election after restart and holds it throughout.
Creation # 1 happens in the fully mixed state (2 old + 2 new nodes). This is the most interesting from an ID generation perspective — both upgraded and non-upgraded nodes are data nodes.
Creations # 2–6 happen in a partially mixed state (1 old node v9.3.3-3 still in cluster).
Creations # 7–15 happen after all old nodes have left — pure 9.4.0 cluster. These are the "fully upgraded" test runs.

EDIT 2: Node v9.3.3-2 and v9.3.3-3 die because of changes in serialization of confidence_interval mapping. It's a known issue:

Tracking issue: [CI] MixedClusterClientYamlTestSuiteIT class failing #143980 — [CI] MixedClusterClientYamlTestSuiteIT class failing (open, assigned to tteofili, labelled priority:high)
• Fix PR: Make int4 confidence_interval BWC for mixed cluster scenarios #144597 — Make int4 confidence_interval BWC for mixed cluster scenarios (open, references [CI] MixedClusterClientYamlTestSuiteIT class failing #143980 as the issue it fixes)

Full stacktrace:

[2026-03-19T15:54:15,799][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v9.3.3-2] fatal error in thread [elasticsearch[v9.3.3-2][clusterApplierService#updateTask][T#1]], exiting
java.lang.AssertionError: provided source [{"_doc":{"properties":{"another_vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100}},"name":{"type":"keyword"},"vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100}}}}}] differs from mapping [{"_doc":{"properties":{"another_vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100,"confidence_interval":0.0}},"name":{"type":"keyword"},"vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100,"confidence_interval":0.0}}}}}]
	at org.elasticsearch.index.mapper.DocumentMapper.<init>(DocumentMapper.java:74) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.MapperService.newDocumentMapper(MapperService.java:644) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:396) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.index.IndexService.updateMapping(IndexService.java:886) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndicesAndUpdateShards(IndicesClusterStateService.java:644) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.doApplyClusterState(IndicesClusterStateService.java:359) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:304) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:605) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:591) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:564) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:493) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:183) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:218) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:184) ~[elasticsearch-9.3.3-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
	at java.lang.Thread.run(Thread.java:1474) ~[?:?]

…etic-id-default-and-remove-feature-flag

This reverts commit 4bc666f.

…ic-id-default-and-remove-feature-flag

tlrx · 2026-04-03T09:13:45Z

Triggering a bunch of CI builds
Green	1267fba	f3283a1
Red			d8c45ff¹	8300538	107f7fe

Known test failures:

https://github.com/elastic/elasticsearch/issues/145421 ↩

…e-feature-flag

kkrik-es · 2026-04-06T15:39:59Z

Let's also add a release highlight note for this.

…etic-id-default-and-remove-feature-flag

docs/changelog/144184.yaml

…e-feature-flag

burqen · 2026-04-07T11:37:51Z

This is getting close to ready. The only thing missing now is a green build and a 👍 from you fine Storage engine folk. I'm pinging you @kkrik-es, since you've been involved here lately, but please feel free to forward the review request if you think that is a better fit.

server/src/main/java/org/elasticsearch/index/IndexSettings.java

…stic#144184) * Synthetic id field for time_series indices This commit removes the feature flag for synthetic id and thereby makes it officially available in production. The setting is controlled by `index.mapping.synthetic_id` which default to `true` for new time_series indices, as long as they use the default codec. Indices with synthetic _id fields doesn't store the id physically on disk. Instead, the id is materialized synthetically from other document fields. This reduces the footprint on disk for time_series indices. * Update docs/changelog/144184.yaml * Temporary logging for tsdb/25_id Add logging for index settings and node versions after index creation in ClientYamlTestClient * Bump IndexVersion for synthetic id On by default Bump the IndexVersion that protect default behavior so that existing time_series indices keep their current behavior and only new indices, created with the new index version use synthetic id by default. * Remove feature flag added in merge * Avoid use of setting when feature flag disabled Only set DISABLE_SEQUENCE_NUMBERS explicitly if the feature flag is enabled in TSDBSyntheticIdsIT. * Disable synthetic id in faulty version range --------- Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com> Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>

burqen requested review from a team as code owners March 13, 2026 11:24

burqen changed the title ~~Ap/2026.03.13.synthetic id default and remove feature flag~~ Remove synthetic id FeatureFlag Mar 13, 2026

burqen marked this pull request as draft March 13, 2026 11:25

elasticsearchmachine added the v9.4.0 label Mar 13, 2026

burqen added >feature :StorageEngine/TSDB You know, for Metrics labels Mar 13, 2026

burqen mentioned this pull request Mar 13, 2026

Enable synthetic id feature, remove feature flag #142366

Closed

16 tasks

burqen added the test-release Trigger CI checks against release build label Mar 13, 2026

burqen force-pushed the ap/2026.03.13.synthetic-id-default-and-remove-feature-flag branch from 2642cff to d0d08f6 Compare March 19, 2026 13:15

burqen changed the title ~~Remove synthetic id FeatureFlag~~ Synthetic id field for time_series indices (remove feature flag) Mar 19, 2026

burqen requested review from fcofdez and tlrx March 19, 2026 13:16

burqen marked this pull request as ready for review March 19, 2026 13:16

elasticsearchmachine added the Team:StorageEngine label Mar 19, 2026

Update docs/changelog/144184.yaml

2d2f281

burqen requested review from kkrik-es, martijnvg and romseygeek March 19, 2026 13:24

burqen and others added 3 commits March 19, 2026 16:26

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

7111d90

…e-feature-flag

Temporary logging for tsdb/25_id

4bc666f

Add logging for index settings and node versions after index creation in ClientYamlTestClient

Merge remote-tracking branch 'upstream/main' into ap/2026.03.13.synth…

e745dd6

…etic-id-default-and-remove-feature-flag

burqen added 3 commits March 20, 2026 15:15

Remove feature flag added added in merge

6c6a6c1

Merge remote-tracking branch 'upstream/main' into ap/2026.03.13.synth…

cb4cbcd

…etic-id-default-and-remove-feature-flag

Revert "Temporary logging for tsdb/25_id"

e207ea1

This reverts commit 4bc666f.

burqen and others added 8 commits April 1, 2026 16:35

Disable synthetic id in faulty version range

4f778a1

Merge remote-tracking branch 'origin/main' into ap/2026.03.13.synthet…

e455a3e

…ic-id-default-and-remove-feature-flag

Merge remote-tracking branch 'origin/main' into ap/2026.03.13.synthet…

e17ab96

…ic-id-default-and-remove-feature-flag

Fix compilation

0ea5365

Merge remote-tracking branch 'origin/main' into ap/2026.03.13.synthet…

c84e0f4

…ic-id-default-and-remove-feature-flag

Merge remote-tracking branch 'origin/main' into ap/2026.03.13.synthet…

320c8f2

…ic-id-default-and-remove-feature-flag

Merge remote-tracking branch 'origin/main' into ap/2026.03.13.synthet…

156fcf7

…ic-id-default-and-remove-feature-flag

Merge mistake

1267fba

tlrx added 5 commits April 3, 2026 11:14

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

f3283a1

…e-feature-flag

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

d8c45ff

…e-feature-flag

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

8300538

…e-feature-flag

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

107f7fe

…e-feature-flag

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

61d2575

…e-feature-flag

burqen added 3 commits April 7, 2026 08:56

Merge remote-tracking branch 'upstream/main' into ap/2026.03.13.synth…

7d66fc5

…etic-id-default-and-remove-feature-flag

Improve release note

50d77a0

Fix issue with release note

19440ea

kkrik-es reviewed Apr 7, 2026

View reviewed changes

docs/changelog/144184.yaml Show resolved Hide resolved

burqen and others added 5 commits April 7, 2026 09:15

Release note adhere to schema

ecb11b0

Remove use of feature flag

a2f7513

Add notable to release note

e892596

Merge branch 'main' into ap/2026.03.13.synthetic-id-default-and-remov…

5c6b5ea

…e-feature-flag

Better release note

e800878

fcofdez requested a review from kkrik-es April 7, 2026 11:31

kkrik-es reviewed Apr 7, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/index/IndexSettings.java Show resolved Hide resolved

kkrik-es approved these changes Apr 7, 2026

View reviewed changes

burqen merged commit 8d84fa9 into elastic:main Apr 7, 2026
37 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synthetic id field for time_series indices (remove feature flag)#144184

Synthetic id field for time_series indices (remove feature flag)#144184
burqen merged 39 commits intoelastic:mainfrom
burqen:ap/2026.03.13.synthetic-id-default-and-remove-feature-flag

burqen commented Mar 13, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Mar 19, 2026

Uh oh!

elasticsearchmachine commented Mar 19, 2026

Uh oh!

burqen commented Mar 19, 2026

Uh oh!

burqen commented Mar 20, 2026 •

edited

Loading

Uh oh!

tlrx commented Apr 3, 2026 •

edited

Loading

Uh oh!

kkrik-es commented Apr 6, 2026

Uh oh!

Uh oh!

burqen commented Apr 7, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

burqen commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 19, 2026

Uh oh!

elasticsearchmachine commented Mar 19, 2026

Uh oh!

burqen commented Mar 19, 2026

Uh oh!

burqen commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Analysis of id mismatch failure in MixedClusterClientYamlTestSuiteIT

Cluster Timeline — id_generation_test Analysis

Setup sequence

Index creation cluster membership

Key observations

Uh oh!

tlrx commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

kkrik-es commented Apr 6, 2026

Uh oh!

Uh oh!

burqen commented Apr 7, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

burqen commented Mar 13, 2026 •

edited

Loading

burqen commented Mar 20, 2026 •

edited

Loading

Cluster Timeline — `id_generation_test` Analysis

tlrx commented Apr 3, 2026 •

edited

Loading