Skip to content

[AWS] Network Firewall logs ingest pipeline duplicate field error #5071

@ottramst

Description

@ottramst

Hey, guys!

Seeing an issue with AWS Network Firewall ingest pipeline specifically on the JSON processor. Using aws package version 1.29.0.

Example AWS Network Firewall log

{"firewall_name":"outbound","availability_zone":"eu-central-1a","event_timestamp":"1674225318","event":{"app_proto":"tls","src_ip":"10.125.94.186","src_port":5302,"event_type":"alert","alert":{"severity":3,"signature_id":4,"rev":0,"signature":"aws:alert_established action","action":"allowed","category":""},"flow_id":1829676028160453,"dest_ip":"123.123.123.123","proto":"TCP","tls":{"subject":"CN=sqs.eu-central-1.amazonaws.com","issuerdn":"C=US, O=Amazon, OU=Server CA 1B, CN=Amazon","subject":"CN=sqs.eu-central-1.amazonaws.com","issuerdn":"C=US, O=Amazon, OU=Server CA 1B, CN=Amazon","serial":"08:C9:76:68:CB:30:31:1C:B2:24:A4:B3:22:F0:16:29","fingerprint":"34:36:32:fb:05:65:83:55:a0:8e:24:7b:41:52:bf:98:88:3b:bf:9b","sni":"sqs.eu-central-1.amazonaws.com","version":"TLS 1.2","notbefore":"2022-11-03T00:00:00","notafter":"2023-10-16T23:59:59","ja3":{},"ja3s":{}},"dest_port":443,"timestamp":"2023-01-20T14:35:18.307095+0000"}}

Ingest pipeline fails with the next error:

{
  "root_cause": [
    {
      "type": "x_content_parse_exception",
      "reason": "[1:502] Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]"
    }
  ],
  "type": "x_content_parse_exception",
  "reason": "[1:502] Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]",
  "caused_by": {
    "type": "json_parse_exception",
    "reason": "Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]"
  }
}

Metadata

Metadata

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions