Context
When a document is sent from Logstash to Elasticsearch for indexing, it can contain an event object, depending on the Logstash configuration and version.
For example, with a document like this one:
{
"message: "...",
"event": {
"original": "...",
"module"=>"azure",
"dataset"=>"azure.activitylogs"
}
}
At lines pipeline.yml#L29-L31 the pipeline tries to rename the field message into event.original and failing because it already exists 1.
The pipeline fails, but since it has an on_failure handler, it stops executing and sends the partially processed document to the mapping stage. The unprocessed fields like claims contain values that do not work well with existing mapping, causing a mapper_parsing_exception.
Steps to reproduce
Requirements:
- Elastic Stack version 8.2.0
- Azure Logs integration version 1.1.8
On your Elastic Stack:
- Install the Azure Logs integration
- Open the Dev Tools in Kibana
- Simulate the pipeline execution using the following request:
POST _ingest/pipeline/logs-azure.activitylogs-1.1.8/_simulate
{
"docs": [
{
"_source": {
"tags": [
"preserve_original_event"
],,
"event": {
"original": """{"time":"2022-05-06T08:48:13.7629029Z", "resourceId":"/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123"}"""
},
"message": """{"time":"2022-05-06T08:48:13.7629029Z", "resourceId":"/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123"}"""
}
}
]
}
- The following response will appear:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_id" : "_id",
"_source" : {
"ecs" : {
"version" : "8.0.0"
},
"event" : {
"original" : """{"time":"2022-05-06T08:48:13.7629029Z", "resourceId":"/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123"}"""
},
"message" : """{"time":"2022-05-06T08:48:13.7629029Z", "resourceId":"/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123"}""",
"error" : {
"message" : "Processor \"rename\" with tag \"\" in pipeline \"\" failed with message \"field [event.original] already exists\""
}
},
"_ingest" : {
"timestamp" : "2022-07-01T13:24:17.846285952Z"
}
}
}
]
}
Expected result
The document is indexed successfully without errors:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_id" : "_id",
"_source" : {
"cloud" : {
"provider" : "azure"
},
"@timestamp" : "2022-05-06T08:48:13.762Z",
"ecs" : {
"version" : "8.0.0"
},
"event" : {
"kind" : "event",
"original" : """{"time":"2022-05-06T08:48:13.7629029Z", "resourceId":"/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123"}""",
},
"azure" : {
"subscription_id" : "6400D8D5-4ACC-478A-8781-13716543D9D2",
"resource" : {
"name" : "TEST123",
"id" : "/SUBSCRIPTIONS/6400D8D5-4ACC-478A-8781-13716543D9D2/RESOURCEGROUPS/ABC123/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/TEST123",
"provider" : "MICROSOFT.STORAGE/STORAGEACCOUNTS",
"group" : "ABC123"
},
"activitylogs" : {
"event_category" : "Administrative"
}
}
},
"_ingest" : {
"timestamp" : "2022-07-01T13:35:52.77762076Z"
}
}
}
]
}
Context
When a document is sent from Logstash to Elasticsearch for indexing, it can contain an
eventobject, depending on the Logstash configuration and version.For example, with a document like this one:
At lines pipeline.yml#L29-L31 the pipeline tries to rename the field
messageintoevent.originaland failing because it already exists 1.The pipeline fails, but since it has an
on_failurehandler, it stops executing and sends the partially processed document to the mapping stage. The unprocessed fields like claims contain values that do not work well with existing mapping, causing amapper_parsing_exception.Steps to reproduce
Requirements:
On your Elastic Stack:
Expected result
The document is indexed successfully without errors:
Footnotes
"If the field doesn't exist or the new name is already used, an exception will be thrown." — the docs. ↩