-
Notifications
You must be signed in to change notification settings - Fork 2.5k
[Feature Request] Add periodic flush for pull-based ingestion #19860
Copy link
Copy link
Closed
Labels
IndexingIndexing, Bulk Indexing and anything related to indexingIndexing, Bulk Indexing and anything related to indexingenhancementEnhancement or improvement to existing feature or requestEnhancement or improvement to existing feature or requestuntriaged
Description
Is your feature request related to a problem? Please describe
OpenSearch currently supports configuring size based thresholds for flush (reference). This does not apply to pull-based ingestion as translog is not used. Instead, add a new time based flush policy to commit the batchStartPointer periodically.
Describe the solution you'd like
Add a time-based flush policy, persisting fresh batchStartPointer in the commit. This should ensure replica recovery does not have to replay too many messages and prevent offset out-of-bounds after exceeding retention period.
Related component
Indexing
Describe alternatives you've considered
No response
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
IndexingIndexing, Bulk Indexing and anything related to indexingIndexing, Bulk Indexing and anything related to indexingenhancementEnhancement or improvement to existing feature or requestEnhancement or improvement to existing feature or requestuntriaged