Skip to content

[AWS] Support VPC flow logs with message field#4830

Merged
kaiyan-sheng merged 5 commits intoelastic:mainfrom
kaiyan-sheng:scrape_message_field_vpcflow
Dec 15, 2022
Merged

[AWS] Support VPC flow logs with message field#4830
kaiyan-sheng merged 5 commits intoelastic:mainfrom
kaiyan-sheng:scrape_message_field_vpcflow

Conversation

@kaiyan-sheng
Copy link
Copy Markdown

@kaiyan-sheng kaiyan-sheng commented Dec 14, 2022 •
edited
Loading

What does this PR do?

When VPC flow logs sent directly from VPC without CloudWatch, they come in with the message field that looks like this:

{\"message\":\"2 428961148399 eni-0e0bf7be352692297 - - - - - - - 1671029698 1671029728 - NODATA\"}

This PR is to add support in the VPC flow log ingest pipeline to support this format.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner December 14, 2022 23:01
@kaiyan-sheng kaiyan-sheng self-assigned this Dec 14, 2022
@kaiyan-sheng kaiyan-sheng added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Dec 14, 2022
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 14, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-12-15T20:49:58.070+0000

  • Duration: 37 min 19 sec

Test stats 🧪

Test Results
Failed 0
Passed 178
Skipped 3
Total 181

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 14, 2022 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (14/14) 💚
Files 93.333% (14/15) 👎 -4.423
Classes 93.333% (14/15) 👎 -4.423
Methods 85.214% (219/257) 👎 -6.144
Lines 96.022% (5914/6159) 👍 4.255
Conditionals 100.0% (0/0) 💚

@andrewkroh
Copy link
Copy Markdown
Member

andrewkroh commented Dec 15, 2022

Could you please elaborate in the PR description about what scenarios lead to this format. What writes the logs in the format and what input is used to read them?

@kaiyan-sheng
Copy link
Copy Markdown
Author

kaiyan-sheng commented Dec 15, 2022

@andrewkroh Please refer to the private github issue that I linked with the PR for more information. Thank you!

kaiyan-sheng
kaiyan-sheng commented Dec 15, 2022
View reviewed changes
packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-with-message-field.log Outdated
Copy link
Copy Markdown
Author

@kaiyan-sheng kaiyan-sheng Dec 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the test case is efficient, I don't think the nested message level actually happens hmmm

The events look like this when VPC flow logs are sent directly without going through CloudWatch.

tommyers-elastic
tommyers-elastic approved these changes Dec 15, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM nice work kaiyan ! is there a way to test the dot expander change?

@kaiyan-sheng
Copy link
Copy Markdown
Author

kaiyan-sheng commented Dec 15, 2022 •
edited
Loading

@tommyers-elastic Thanks. I didn't find a way to add test into for example _dev/test/pipeline but we can test it manually using the Kibana dev tools simulate. For example:

POST _ingest/pipeline/logs-aws.vpcflow-1.28.2/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "2 123456 eni-0c1c3a5c0ac2c4c95 162.142.125.229 172.31.29.121 5583 6352 6 1 44 1648766030 1648766085 REJECT OK",
        "@timestamp": "2022-03-31T22:33:50Z",
        "event.id": "36768711127022897352489210216814512733652448397667467264",
        "cloud.provider": "aws",
        "cloud.account.id": "428152502467",
        "cloud.region": "us-east-1",
        "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456:deliverystream/vpc-flow-log-stream-http-endpoint",
        "aws.firehose.request_id": "a70eae51-0629-4c78-b0dc-799a19642585",
        "aws.kinesis.type": "deliverystream",
        "aws.kinesis.name": "vpc-flow-log-stream-http-endpoint"
      }
    }
    ]
}

Without the dot_expander processor, the output will have cloud.provider twice, one flat and one nested.

@kaiyan-sheng kaiyan-sheng merged commit 2d031ca into elastic:main Dec 15, 2022
@kaiyan-sheng kaiyan-sheng deleted the scrape_message_field_vpcflow branch December 15, 2022 22:09
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 15, 2022

Package aws - 1.28.2 containing this change is available at https://epr.elastic.co/search?package=aws

1 similar comment
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 15, 2022

Package aws - 1.28.2 containing this change is available at https://epr.elastic.co/search?package=aws

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tommyers-elastic tommyers-elastic tommyers-elastic approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaiyan-sheng @elasticmachine @andrewkroh @tommyers-elastic