Azure is a bit inconsistent in the format of activity logs. I have found that sometimes, a field that is usually json is instead a string containng serialized json. When such a log is present, the filebeat activity log ingest pipeline fails to parse it, thus resulting in a mapping_parser_exception.
For one log example I have created a discuss thread, see below. I have also found another type of log where this happens. I have been in contact with support and have gotten a go ahead to create a PR. So I am now creating this issue to be able to connect the PR to this.
{
"_source": {
"agent": {
"ephemeral_id": "myid",
"hostname": "myhost",
"id": "id",
"name": "filebeat-64b5dc8949-md5p9",
"type": "filebeat",
"version": "7.11.1"
},
"azure": {
"consumer_group": "filebeat",
"enqueued_time": "2021-03-25T09:27:44.332Z",
"eventhub": "myeventhub",
"offset": 2147509126360,
"sequence_number": 2773145
},
"cloud": {
"account": {},
"instance": {
"id": "myid",
"name": "myname"
},
"machine": {
"type": "Standard_E8s_v3"
},
"provider": "azure",
"region": "westeurope"
},
"ecs": {
"version": "1.7.0"
},
"event": {
"dataset": "azure.activitylogs",
"module": "azure"
},
"fileset": {
"name": "activitylogs"
},
"input": {
"type": "azure-eventhub"
},
"message": "{\"Authorization\":\"null\",\"Claims\":\"{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress\\\":\\\"Microsoft.RecoveryServices\\\"}\",\"DeploymentUnit\":\"myunit\",\"EventId\":162,\"EventName\":\"AzureBackupActivityLog\",\"ResultDescription\":\"Backup Succeeded\",\"category\":\"Administrative\",\"correlationId\":\"111a1aa1-a1a1-1aa1-a111-1a11aa1aa111\",\"durationMs\":0,\"eventName\":\"Backup\",\"identity\":\"{\\\"claims\\\":{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress\\\":\\\"Microsoft.RecoveryServices\\\"}}\",\"level\":\"Informational\",\"location\":\"westeurope\",\"operationId\":\"aa111a11-1a11-111a-11a1-1ad11a1a1a11\",\"operationName\":\"Microsoft.RecoveryServices/vaults/backupFabrics/protectionContainers/protectedItems/backup/action\",\"operationVersion\":\"null\",\"properties\":{\"Entity Name\":\"mymachine\",\"Job Id\":\"jobid\",\"Start Time\":\"2021-03-24 21:50:40Z\"},\"resourceId\":\"/SUBSCRIPTIONS/111A1A11-11A1-1AAA-1111-11AAA111AAAA/RESOURCEGROUPS/MY-GROUP/PROVIDERS/MICROSOFT.RECOVERYSERVICES/VAULTS/RSV-DATAPROTECTION\",\"resultType\":\"Succeeded\",\"time\":\"2021-03-25T09:22:28.4002017Z\"}",
"service": {
"type": "azure"
},
"tags": [
"forwarded"
]
}
},
{
"_source": {
"agent": {
"ephemeral_id": "111aaa11-a111-1aaa-aa11-a11aa1a11111",
"hostname": "filebeat-64b5dc8949-n29fn",
"id": "a111a1aa-1111-11aa-a1a1-1aaa111111aa",
"name": "filebeat-64b5dc8949-n29fn",
"type": "filebeat",
"version": "7.11.1"
},
"azure": {
"consumer_group": "filebeat",
"enqueued_time": "2021-05-03T07:37:07.67Z",
"eventhub": "myeventhub",
"offset": 2263463355376,
"sequence_number": 2984235
},
"cloud": {
"account": {},
"instance": {
"id": "aa1111a1-aa11-11aa-a1aa-a11a1aaa111",
"name": "myinstance"
},
"machine": {
"type": "Standard_E8s_v3"
},
"provider": "azure",
"region": "westeurope"
},
"ecs": {
"version": "1.7.0"
},
"event": {
"dataset": "azure.activitylogs",
"module": "azure"
},
"fileset": {
"name": "activitylogs"
},
"input": {
"type": "azure-eventhub"
},
"message": "{\"category\":\"Security\",\"correlationId\":\"11aa11aa-a111-1111-a11a-aaa1aaa1aa1a\",\"description\":\"Microsoft system deleted the 1111a111-11aa-1aa1-a1aa-111a11111a11 subscription.\",\"identity\":\"{\\\"authorization\\\":{},\\\"claims\\\":{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/spn\\\":\\\"microsoft system\\\"}}\",\"operationName\":\"Delete subscription.\",\"properties\":\"{ \\\"message\\\": \\\"Microsoft system deleted the 1111a111-11aa-1aa1-a1aa-111a11111a11 subscription.\\\"}\",\"resourceId\":\"/SUBSCRIPTIONS/1111A111-11AA-1AA1-A1AA-111A11111A11\",\"resultType\":\"Succeeded\",\"time\":\"2021-05-03T07:35:54.1778906Z\"}",
"service": {
"type": "azure"
},
"tags": [
"forwarded"
]
}
}
]
Azure is a bit inconsistent in the format of activity logs. I have found that sometimes, a field that is usually json is instead a string containng serialized json. When such a log is present, the filebeat activity log ingest pipeline fails to parse it, thus resulting in a mapping_parser_exception.
For one log example I have created a discuss thread, see below. I have also found another type of log where this happens. I have been in contact with support and have gotten a go ahead to create a PR. So I am now creating this issue to be able to connect the PR to this.
For confirmed bugs, please report:
Simulate the filebeat--activitylogs-pipeline with one of the following documents. The pipeline will throw an error.
{ "_source": { "agent": { "ephemeral_id": "myid", "hostname": "myhost", "id": "id", "name": "filebeat-64b5dc8949-md5p9", "type": "filebeat", "version": "7.11.1" }, "azure": { "consumer_group": "filebeat", "enqueued_time": "2021-03-25T09:27:44.332Z", "eventhub": "myeventhub", "offset": 2147509126360, "sequence_number": 2773145 }, "cloud": { "account": {}, "instance": { "id": "myid", "name": "myname" }, "machine": { "type": "Standard_E8s_v3" }, "provider": "azure", "region": "westeurope" }, "ecs": { "version": "1.7.0" }, "event": { "dataset": "azure.activitylogs", "module": "azure" }, "fileset": { "name": "activitylogs" }, "input": { "type": "azure-eventhub" }, "message": "{\"Authorization\":\"null\",\"Claims\":\"{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress\\\":\\\"Microsoft.RecoveryServices\\\"}\",\"DeploymentUnit\":\"myunit\",\"EventId\":162,\"EventName\":\"AzureBackupActivityLog\",\"ResultDescription\":\"Backup Succeeded\",\"category\":\"Administrative\",\"correlationId\":\"111a1aa1-a1a1-1aa1-a111-1a11aa1aa111\",\"durationMs\":0,\"eventName\":\"Backup\",\"identity\":\"{\\\"claims\\\":{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress\\\":\\\"Microsoft.RecoveryServices\\\"}}\",\"level\":\"Informational\",\"location\":\"westeurope\",\"operationId\":\"aa111a11-1a11-111a-11a1-1ad11a1a1a11\",\"operationName\":\"Microsoft.RecoveryServices/vaults/backupFabrics/protectionContainers/protectedItems/backup/action\",\"operationVersion\":\"null\",\"properties\":{\"Entity Name\":\"mymachine\",\"Job Id\":\"jobid\",\"Start Time\":\"2021-03-24 21:50:40Z\"},\"resourceId\":\"/SUBSCRIPTIONS/111A1A11-11A1-1AAA-1111-11AAA111AAAA/RESOURCEGROUPS/MY-GROUP/PROVIDERS/MICROSOFT.RECOVERYSERVICES/VAULTS/RSV-DATAPROTECTION\",\"resultType\":\"Succeeded\",\"time\":\"2021-03-25T09:22:28.4002017Z\"}", "service": { "type": "azure" }, "tags": [ "forwarded" ] } }, { "_source": { "agent": { "ephemeral_id": "111aaa11-a111-1aaa-aa11-a11aa1a11111", "hostname": "filebeat-64b5dc8949-n29fn", "id": "a111a1aa-1111-11aa-a1a1-1aaa111111aa", "name": "filebeat-64b5dc8949-n29fn", "type": "filebeat", "version": "7.11.1" }, "azure": { "consumer_group": "filebeat", "enqueued_time": "2021-05-03T07:37:07.67Z", "eventhub": "myeventhub", "offset": 2263463355376, "sequence_number": 2984235 }, "cloud": { "account": {}, "instance": { "id": "aa1111a1-aa11-11aa-a1aa-a11a1aaa111", "name": "myinstance" }, "machine": { "type": "Standard_E8s_v3" }, "provider": "azure", "region": "westeurope" }, "ecs": { "version": "1.7.0" }, "event": { "dataset": "azure.activitylogs", "module": "azure" }, "fileset": { "name": "activitylogs" }, "input": { "type": "azure-eventhub" }, "message": "{\"category\":\"Security\",\"correlationId\":\"11aa11aa-a111-1111-a11a-aaa1aaa1aa1a\",\"description\":\"Microsoft system deleted the 1111a111-11aa-1aa1-a1aa-111a11111a11 subscription.\",\"identity\":\"{\\\"authorization\\\":{},\\\"claims\\\":{\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/spn\\\":\\\"microsoft system\\\"}}\",\"operationName\":\"Delete subscription.\",\"properties\":\"{ \\\"message\\\": \\\"Microsoft system deleted the 1111a111-11aa-1aa1-a1aa-111a11111a11 subscription.\\\"}\",\"resourceId\":\"/SUBSCRIPTIONS/1111A111-11AA-1AA1-A1AA-111A11111A11\",\"resultType\":\"Succeeded\",\"time\":\"2021-05-03T07:35:54.1778906Z\"}", "service": { "type": "azure" }, "tags": [ "forwarded" ] } } ]