The mutate processor should be able to modify the meta data (index, type, id etc.). This should be configured in the meta section of the mutate processor. The following meta data properties should be able to be modified in the meta section: _index, _type, _id, _routing, _parent, _timestamp and _ttl.

Mustache templates should be used in order to refer to actual field values in the documents being processed.

For example:

{
   "meta" : {
      "_index" : "mycompany-{{department}}"
   }
}

The index being index to would depend on the department field in a document being processed.

Pipeline example:

{
   "description" : "...",
   "processors" : [
      {
         "grok" : {
            "field" : "message",
            "pattern": "%{IPORHOST:destination} %{DATESTAMP:timestamp} %{DATA:message}"
         }
      },
      {
         "mutate" : {
            "meta": {
               "_index": "logs-{{destination}}",
               "_timestamp": "{{timestamp}}"
            },
            ...
         }
      }
   ]
}

In the above example the destination and the timestamp properties extracted by the grok processor are used by the mutate processor to determine to what index to send the log to and to attach the timestamp to the _timestamp meta field.

For now the plan is to add template support only to the meta options of the mutate processor, but the idea is that later almost any processor and pipeline option should be temptable.

Open question remains if all processors should be able to modify the metadata of a document.