-
Notifications
You must be signed in to change notification settings - Fork 313
Make opensearch source scroll timeout configurable and increase default value #5679
Copy link
Copy link
Closed
Description
Is your feature request related to a problem? Please describe.
The current default for the scroll timeout to process a batch is 1 minute and this is not configurable (
Line 52 in b422250
| static final String SCROLL_TIME_PER_BATCH = "1m"; |
If this times out due to buffer being full or circuit breaker blocking writes to the buffer, it will result in this exception and reprocessing will take place for the index
025-05-05T16:10:34.168 [pool-12-thread-1] ERROR org.opensearch.dataprepper.plugins.source.opensearch.worker.ScrollWorker - Unknown exception while processing index 'my-index':
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [search_phase_execution_exception] all shards failed
Describe the solution you'd like
Let this be configurable under search_options
search_options:
search_context_type: "scroll"
scroll_time_per_batch: "15m" // increase default to 10m
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
Status
Done