Skip to content

Make int4 confidence_interval BWC for mixed cluster scenarios#144597

Closed
tteofili wants to merge 4 commits intoelastic:mainfrom
tteofili:int4_ci_bwc
Closed

Make int4 confidence_interval BWC for mixed cluster scenarios#144597
tteofili wants to merge 4 commits intoelastic:mainfrom
tteofili:int4_ci_bwc

Conversation

@tteofili
Copy link
Copy Markdown
Contributor

@tteofili tteofili commented Mar 19, 2026

This should be the one fix for #143980

@elasticsearchmachine elasticsearchmachine added v9.4.0 Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Mar 19, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

elasticsearchmachine commented Mar 19, 2026

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@davidkyle
Copy link
Copy Markdown
Member

davidkyle commented Mar 20, 2026

The problem appears to be that a previous version confidenceInterval defaulted to 0.0f which meant it passed the null check in toXContent and was always written.

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

Line 2038 in 194133b

this.confidenceInterval = confidenceInterval == null ? 0f : confidenceInterval;

So the failure in #143980 is because the old node writes confidenceInterval and the new one doesn't?

Now confidence interval is deprecated we don't want the default value to been seen but that modifies the representation of the mapping. I think hiding confidenceInterval can only be done with a NodeFeature

@davidkyle davidkyle added >test Issues or PRs that are addressing/adding tests test-full-bwc Trigger full BWC version matrix tests and removed >non-issue labels Mar 20, 2026
@davidkyle
Copy link
Copy Markdown
Member

davidkyle commented Mar 20, 2026

The bwc-tests on this PR failed with the same error. The assertion failed on the upgraded node

java.lang.AssertionError: provided source [{"_doc":{"properties":{"another_vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_flat","confidence_interval":0.0}},"name":{"type":"keyword"},"vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_flat","confidence_interval":0.0}}}}}] differs from mapping [{"_doc":{"properties":{"another_vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_flat"}},"name":{"type":"keyword"},"vector":{"type":"dense_vector","dims":4,"index":true,"similarity":"l2_norm","index_options":{"type":"int4_flat"}}}}}]

@davidkyle
Copy link
Copy Markdown
Member

davidkyle commented Mar 20, 2026

Looking at the assertion in DocumentMapper I think this problem has occurred before and was addressed by the isSyntheticSourceMalformed clause added in #98808

https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java#L77

Adding that leniency would work for upgraded nodes but in one of the failures it was the older node that threw (9.2.8 in the stack trace below). Perhaps adding cluster feature check and always writing confidence_interval: 0.0 if null in a mixed cluster then in a upgraded cluster that write can be avoided with an extra clause on the assertion.

[2026-03-20T05:51:02,816][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v9.2.8-2] fatal error in thread [elasticsearch[v9.2.8-2][clusterApplierService#updateTask][T#1]], exiting
java.lang.AssertionError: provided source [{"_doc":{"properties":{"embedding":{"type":"dense_vector","dims":4,"index":true,"similarity":"cosine","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100}}}}}] differs from mapping [{"_doc":{"properties":{"embedding":{"type":"dense_vector","dims":4,"index":true,"similarity":"cosine","index_options":{"type":"int4_hnsw","m":16,"ef_construction":100,"confidence_interval":0.0}}}}}]
	at org.elasticsearch.index.mapper.DocumentMapper.<init>(DocumentMapper.java:74) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.MapperService.newDocumentMapper(MapperService.java:612) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:364) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.index.IndexService.updateMapping(IndexService.java:885) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]c
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.updateIndices(IndicesClusterStateService.java:704) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.doApplyClusterState(IndicesClusterStateService.java:357) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:304) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:572) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:558) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:531) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:460) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:159) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:218) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:184) ~[elasticsearch-9.2.8-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
	at java.lang.Thread.run(Thread.java:1474) ~[?:?]

@benwtrent benwtrent mentioned this pull request Mar 20, 2026
Closed
@burqen burqen mentioned this pull request Mar 23, 2026
Merged
@tteofili
Copy link
Copy Markdown
Contributor Author

tteofili commented Mar 25, 2026

superseeded by #144739

@tteofili tteofili closed this Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benwtrent benwtrent benwtrent left review comments

Assignees

No one assigned

Labels

:Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch >test Issues or PRs that are addressing/adding tests test-full-bwc Trigger full BWC version matrix tests v9.4.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tteofili @elasticsearchmachine @davidkyle @benwtrent