-
Notifications
You must be signed in to change notification settings - Fork 2.5k
[RFC] Support System Generated Ingest Pipeline/Processor #17509
Description
Is your feature request related to a problem? Please describe
I'm working on a proposal in neural plugin to simply the neural search set up. We want to remove the step that user needs to set up an ingest pipeline to use ML model to generate the embedding.
Describe the solution you'd like
We propose to create a new field type semantic for original data. Then during indexing OpenSearch will check if there is a semantic field. If there is one it will automatically create an ingest processor and append it to the final ingest pipeline. If there is no final ingest pipeline then we will create a pipeline with that processor as the final ingest pipeline. This auto generated ingest processor is invisible to users and they don't need to manage it. In this solution we auto generate the ingest processor only based on the index configuration and we will limit the scope to that.
But we are also thinking should we set up a more generic solution for system generated ingest pipelines/processors for use cases that we want to auto generate them to simplify the user experience?
Related component
No response
Describe alternatives you've considered
No response
Additional context
[RFC] Support Semantic Field Type to Simplify Neural Search Set Up HLD
[RFC] Support Semantic Field Type to Simplify Neural Search Set Up LLD