Skip to content

VPC Flow Log header line forwarded to Datadog as log entry #1043

@rohanpower

Description

@rohanpower

When processing VPC Flow Logs from S3, the Datadog Forwarder forwards the header line as a regular log entry to Datadog instead of skipping it.

Expected Behavior:
VPC Flow Log header lines (e.g. version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status) should be skipped and not forwarded to Datadog.

Actual Behavior:
The header line appears in Datadog as a log entry with the message field containing the header text.

Example Log in Datadog:

{
  "message": "version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status",
  "ddsource": "vpc",
  "service": "vpc"
}

Root Cause:
In 'aws/logs_monitoring/steps/handlers/s3_handler.py', the _extract_other_logs() method processes all lines from VPC Flow Log files without
detecting or skipping the header line.

Proposed Solution:
Detect VPC Flow Logs by checking if "vpcflowlogs" is in the S3 key path, then skip the first line (header) for these files.

Code Location:

  • File: aws/logs_monitoring/steps/handlers/s3_handler.py
  • Function: _extract_other_logs()

Suggested Fix:

is_vpc_flowlog = "vpcflowlogs" in self.data_store.key.lower()

for idx, line in enumerate(self.data_store.data.splitlines()):
    line = line.decode("utf-8", errors="ignore").strip()
    if len(line) == 0:
        continue

    # Skip first line of VPC Flow Logs (header)
    if is_vpc_flowlog and idx == 0:
        continue

    yield self._format_event(line)

Environment:

  • Forwarder Version: 5.1.1
  • S3 Key Pattern: AWSLogs/{account-id}/vpcflowlogs/{region}/{date}/...

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions