• Version: 7.11.1 ( But also had this issue on 7.11.0)
• Operating System: Ubuntu (AKS Cluster)
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-error-while-parsing-lasterror-field-in-azure-platform-logs/265940

Hi everyone,

so I was testing the Azure module for filebeat to pull the logs into ES and run into the following issue. Basically, if there is any error message present in the APIM(API Management service) logs, those will not reach the ES backend. In other words, when the request is not successful on the APIM side, the service can add 6 extra fields to its log entry which are : LastErrorElaspsed, LastErrorSource, LastErrorSection, LasErrorReason, LastErrorMessage, LastErrorScope.

These are the fields whose data is being lost. However, they are being sent by azure as shown by the printed entry in the Filebeat logs:

{
    Fields: {
        "agent": {
            "type": "filebeat",
            "version": "7.11.1"
        },
        "ecs": {
            "version": "1.6.0"
        },
        "event": {
            "dataset": "azure.platformlogs",
            "module": "azure"
        },
        "fileset": {
            "name": "platformlogs"
        },
        "input": {
            "type": "azure-eventhub"
        },
        "message": "{\"Level\":4,\"callerIpAddress\":\"ipHere\",\"category\":\"GatewayLogs\",\"correlationId\":\"idHere\",\"durationMs\":0,\"isRequestSuccess\":false,\"location\":\"North Central US\",\"operationName\":\"Microsoft.ApiManagement/GatewayLogs\",\"properties\":{\"apiId\":\"api-id\",\"apiRevision\":\"1\",\"cache\":\"none\",\"clientProtocol\":\"HTTP/1.1\",\"clientTlsVersion\":\"1.2\",\"lastError\":{\"message\":\"Claim value mismatch: email=sampleemail@something.com.\",\"path\":\"choose\\\\when[1]\",\"reason\":\"TokenClaimValueMismatch\",\"scope\":\"api\",\"section\":\"inbound\",\"source\":\"validate-jwt\"},\"method\":\"GET\",\"operationId\":\"GET\",\"responseCode\":401,\"responseSize\":316,\"url\":\"https://some.url.here\"},\"resourceId\":\"/resourceIdHere\",\"time\":\"2021-03-02T10:15:00.2443029Z\"}",
        "service": {
            "type": "azure"
        }
    }
}

The following warning message accompanied the log entry shown above

2021-03-02T10:16:45.987Z	WARN	[elasticsearch]	elasticsearch/client.go:408	Cannot index event publisher.Event {azure log content here with lastError field included},  Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [azure.platformlogs.properties.lastError] of type [keyword] in document with id 'qLxw8ncBPAyIKXCBqjeG'. Preview of field's value: '{path=choose\\when[1], reason=TokenClaimValueMismatch, scope=api, section=inbound, source=validate-jwt, message=Claim value mismatch: email=sampleemail@something.com.}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1127"}}

These are the steps to reproduce the issue:
1: Use the Filebeat Azure Module to pull platform logs from Azure using the following config:

filebeat:
  modules:
  - module: azure
    platformlogs:
      enabled: true
      var:
        eventhub: "eventhub"
        consumer_group: "consumer"
        connection_string: "connstring"
        storage_account: "storageaccount"
        storage_account_key: "key"

2. Send APIM logs to the eventhub filebeat is reading from
3. Configure the APIM such a way that a request will fail. (ex: token validation failure)
4. Check logs in kibana. The usual log fields should be prensent, but not the LastError fields.

Just a few extra details about my setup. I'm using the kubernetes operator to deploy ES, Kibana and Filebeat. They are all running in an AKS cluster.