Skip to content

[Feature Request] Add periodic flush for pull-based ingestion #19860

@varunbharadwaj

Description

@varunbharadwaj

Is your feature request related to a problem? Please describe

OpenSearch currently supports configuring size based thresholds for flush (reference). This does not apply to pull-based ingestion as translog is not used. Instead, add a new time based flush policy to commit the batchStartPointer periodically.

Describe the solution you'd like

Add a time-based flush policy, persisting fresh batchStartPointer in the commit. This should ensure replica recovery does not have to replay too many messages and prevent offset out-of-bounds after exceeding retention period.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Labels

IndexingIndexing, Bulk Indexing and anything related to indexingenhancementEnhancement or improvement to existing feature or requestuntriaged

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions