Sequence numbers have considerable storage footprint in TSDB, especially for OTel. We propose to rely on merges to remove them when they are no longer needed for peer recoveries and CCR. This disables OCC and changes the behavior of any feature relying on the latter, such as update- and delete-by-query.

This effort requires the following work items:

Distributed:

• Add a new final index setting to enable/disable this mode of operation. Add index.disable_sequence_numbers setting #143221
• Adapt RecoverySourcePruneMergePolicy to account for sequence numbers and remove them during merges. Ensure that the retained doc ids align for the soft deleted documents too to avoid inconsistencies. Prune sequence numbers during merges #143583
• [@tlrx] Add integration tests for peer recoveries and CCR after sequence numbers have been merged away. [Test] Tests sequence number pruning after merge for standard indices #143690 [Test] Test that sequence numbers are not pruned with retention lease #143825 [Test] Randomly disable sequence numbers in CcrTimeSeriesDataStreamsIT #143930 Fix FollowingEngine#lookupPrimaryTerm when sequence numbers are disabled #143935 Adjust ESIntegTestCase.getLiveDocs method to account for pruned sequence numbers #143999 [Test] Add CCR integration tests for pruned sequence numbers #144013 [Test] Test peer-recovery with sequence numbers pruning #144116 Fix replica writes after _seq_no doc values are pruned #144180
• Reject OCC operations when the setting is enabled. Reject OCC and updates on seq-no-disabled indices #143746
• [@fcofdez] Document OCC limitations and new behaviours Add docs for index.disable_sequence_numbers setting #144353
• Adjust the index bulk requests in the update by query machinery to stop using OCC and provide the read _id to validate that the script is not changing the dimensions and the index operation would be indeed an overwrite. Disable OCC in update/delete-by-query for seq_no-less indices #143465
• [ ] Implement a lightweight validation in the Engine to ensure that Index operations are overwriting an existing document.
  • This is not possible to guarantee with INDEX operations since those can target non existing documents and we have weak guarantees anyway, so we'll keep the current behaviour and if the document was deleted between the update by query search and the bulk request we'll recreate the document. We'll document this.
• [@tlrx] Implement a new method to compute the min retained sequence number in serverless to avoid retaining sequence numbers for too long. Skip peer-recovery retention in soft deletes policy for serverless #144223
• Adapt delete by query to stop using OCC Disable OCC in update/delete-by-query for seq_no-less indices #143465 Remove upfront OCC resolution for bulk-by-scroll and handle seq_no per index #144309
• [ ] Evaluate if retention query can be executed once for both RecoverySourcePruneMergePolicy and SoftDeletesRetentionMergePolicy (see comment)
  • I gave it a try and it adds substantial complexities to the existing code for little benefits (the queries themselves are not that costly).
• [ ] Add coverage for disabling sequence numbers in TSDBSyntheticIdsIT following [TEST] Add coverage for disabling sequence numbers in TSDB ITs #143897 (see comment here)

Storage Engine:

• Reject searches that set seq_no_primary_term. Reject search requests with seq_no_primary_term when sequence numbers are disabled #143759
• Reject searches that query a seq_no range of documents. Bar searching or sorting on _seq_no when disabled #143600
• Return sentinel values in bulk responses when the setting is enabled. Remove seq_no from bulk response when disabled #143753
• Return sentinel values to get responses when the setting is enabled. Return sentinel values from GET calls when sequence numbers are disabled #143575
• Adapt ES819TSDBDocValuesFormat to support merging away sequence numbers, as currently it applies some optimizations that are incompatible with the way that the custom merge policy wraps the sequence number readers (note: depends on the 1st Distributed task). (Reproducer: [Draft] Reproducer for TSDB issue with sequence numbers disabled and ES819TSDBDocValuesConsumer #143949) Fix: Don't use optimized merge for fields that will be pruned by the merge #144000
• Allow index operations for backing indices with this setting enabled. Allow index requests with explicit ids on seq_no disabled backing indices #144041
• Reject uses of _seq_no in random_source scoring function and stop using it as a default field when none is supplied. (RandomScoreFunction requires a defined field with disabled _seq_no #144046)
• [@kkrik-es] Execute benchmarks to ensure that eventually the sequence number storage overhead converges toward 0. (Test disabling sequence numbers in TSDB #144043)
• Verify that nested docs still work since we keep the primary term. (Add nested query checks for disabled sequence numbers #144185)
• [ ] Consider enabling this setting randomly in tests and only disable it in tests that rely on OCC, seq nos, etc?
  • Means adding exclusions for every test that uses an update, which with randomization risks causing a lot of churn in CI, test mutes, etc. We think the TSDB coverage is good enough.
• [@kkrik-es] Add coverage for disabling sequence numbers in tsdb IT suites ([TEST] Add coverage for disabling sequence numbers in TSDB ITs #143897)
• [@kkrik-es] Add random merges to existing time series test suites. ([TEST] Add random force merges to TSDB integration tests #145024)