-
Notifications
You must be signed in to change notification settings - Fork 395
Description
When processing VPC Flow Logs from S3, the Datadog Forwarder forwards the header line as a regular log entry to Datadog instead of skipping it.
Expected Behavior:
VPC Flow Log header lines (e.g. version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status) should be skipped and not forwarded to Datadog.
Actual Behavior:
The header line appears in Datadog as a log entry with the message field containing the header text.
Example Log in Datadog:
{
"message": "version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status",
"ddsource": "vpc",
"service": "vpc"
}
Root Cause:
In 'aws/logs_monitoring/steps/handlers/s3_handler.py', the _extract_other_logs() method processes all lines from VPC Flow Log files without
detecting or skipping the header line.
Proposed Solution:
Detect VPC Flow Logs by checking if "vpcflowlogs" is in the S3 key path, then skip the first line (header) for these files.
Code Location:
- File: aws/logs_monitoring/steps/handlers/s3_handler.py
- Function: _extract_other_logs()
Suggested Fix:
is_vpc_flowlog = "vpcflowlogs" in self.data_store.key.lower()
for idx, line in enumerate(self.data_store.data.splitlines()):
line = line.decode("utf-8", errors="ignore").strip()
if len(line) == 0:
continue
# Skip first line of VPC Flow Logs (header)
if is_vpc_flowlog and idx == 0:
continue
yield self._format_event(line)
Environment:
- Forwarder Version: 5.1.1
- S3 Key Pattern: AWSLogs/{account-id}/vpcflowlogs/{region}/{date}/...