Skip to content

Applying destination index pipelines after a processor changes the target index  #57968

@tvernum

Description

@tvernum

When evaluating ingest pipelines, we determine the full set of pipelines upfront (taking into account default_pipeline and final_pipeline) based on the index name that is included in the incoming request.

While executing those pipelines it is possible that a processor will change the destination index , for example (trivial example)

PUT /_ingest/pipeline/redirect
{
  "description": "Redirect to another index",
  "processors": [{ "set": { "field": "_index", "value": "new-destination" } }]
}

If that happens it is possible that the new destination index has its own set of default/final pipelines, but we do not execute them.

It would be preferable if the pipelines attached to the new destination were executed, although there are some questions/issues to work through

  • if the default_pipeline redirects to a new index, should we still execute the final_pipeline of the original index?
  • should we execute the default_pipeline of the new destination, or just the final_pipeline?
  • how do we prevent the document from being infinitely redirected between indices (could we set a rule that the pipeline of the destination index may not change the destination a second time?)

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions