Skip to content

Make opensearch source scroll timeout configurable and increase default value #5679

@graytaylor0

Description

@graytaylor0

Is your feature request related to a problem? Please describe.
The current default for the scroll timeout to process a batch is 1 minute and this is not configurable (

),

If this times out due to buffer being full or circuit breaker blocking writes to the buffer, it will result in this exception and reprocessing will take place for the index

025-05-05T16:10:34.168 [pool-12-thread-1] ERROR org.opensearch.dataprepper.plugins.source.opensearch.worker.ScrollWorker - Unknown exception while processing index 'my-index':
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [search_phase_execution_exception] all shards failed

Describe the solution you'd like
Let this be configurable under search_options

search_options:
  search_context_type: "scroll"
  scroll_time_per_batch: "15m" // increase default to 10m

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status

Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions