Allow index requests with explicit ids on seq_no disabled backing indices#144041
Merged
fcofdez merged 8 commits intoelastic:mainfrom Mar 13, 2026
Merged
Conversation
When sequence numbers are disabled on a data stream backing index, OCC (if_seq_no/if_primary_term) cannot be used. This relaxes the append-only restriction for INDEX ops with an explicit document ID targeting backing indices where seq_no is disabled, either on the target index itself or on any sibling backing index in the data stream.
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
fcofdez
commented
Mar 11, 2026
| } | ||
|
|
||
| static boolean hasAnyBackingIndexWithSeqNoDisabled(DataStream dataStream, Function<Index, IndexMetadata> indexMetadataProvider) { | ||
| for (Index backingIndex : dataStream.getIndices()) { |
Contributor
Author
There was a problem hiding this comment.
I'm exploring the idea of adding a flag into the data stream to tell whether or not it has any backing index with disabled seq nos.
19 tasks
…q-no-disabled-backing-indices
fcofdez
commented
Mar 11, 2026
|
|
||
| @After | ||
| public void resetSearchInterceptor() { | ||
| public void cleanupInterceptors() { |
Contributor
Author
There was a problem hiding this comment.
I hit a failure in CI with this test and I noticed that the request interceptors weren't cleared after each test, so I went ahead and restricted more when we add the assertions based on the index name and added the clearing between runs to avoid any issues. Happy to extract this into another PR if it's adding noise.
tlrx
approved these changes
Mar 12, 2026
| // to handle mixed data streams where OCC is disabled for the entire operation. | ||
| if (writeRequest.id() != null && IndexSettings.DISABLE_SEQUENCE_NUMBERS_FEATURE_FLAG) { | ||
| IndexMetadata targetMetadata = indexMetadataProvider.apply(indexAbstraction.getWriteIndex()); | ||
| if ((targetMetadata != null && IndexSettings.DISABLE_SEQUENCE_NUMBERS.get(targetMetadata.getSettings())) |
Member
There was a problem hiding this comment.
I think we can use targetMetadata.sequenceNumbersDisabled() here? That would be much more efficient.
| static boolean hasAnyBackingIndexWithSeqNoDisabled(DataStream dataStream, Function<Index, IndexMetadata> indexMetadataProvider) { | ||
| for (Index backingIndex : dataStream.getIndices()) { | ||
| IndexMetadata im = indexMetadataProvider.apply(backingIndex); | ||
| if (im != null && IndexSettings.DISABLE_SEQUENCE_NUMBERS.get(im.getSettings())) { |
| var overwriteResponse = client().prepareBulk() | ||
| .add(prepareIndex(backingIndex).setId(createdId).setSource(Map.of("@timestamp", "2020-01-01", "value", "overwritten"))) | ||
| .get(); | ||
| assertNoFailures(overwriteResponse); |
Member
There was a problem hiding this comment.
Should we assert the UPDATED status?
|
|
||
| @After | ||
| public void resetSearchInterceptor() { | ||
| public void cleanupInterceptors() { |
…q-no-disabled-backing-indices
Contributor
Author
|
Thanks for the review Tanguy! |
michalborek
pushed a commit
to michalborek/elasticsearch
that referenced
this pull request
Mar 23, 2026
…ices (elastic#144041) When sequence numbers are disabled on a data stream backing index, OCC (if_seq_no/if_primary_term) cannot be used. This relaxes the append-only restriction for INDEX ops with an explicit document ID targeting backing indices where seq_no is disabled, either on the target index itself or on any sibling backing index in the data stream. Relates elastic#136305
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When sequence numbers are disabled on a data stream backing index,
OCC (if_seq_no/if_primary_term) cannot be used. This relaxes the
append-only restriction for INDEX ops with an explicit document id
targeting backing indices where seq_no is disabled, either on the
target index itself or on any sibling backing index in the data stream.