Skip to content

[Filebeat] Allow - for source IP for AWS S3 Access pipeline#26940

Merged
kaiyan-sheng merged 2 commits intoelastic:masterfrom
legoguy1000:26913-awss3-update
Jul 19, 2021
Merged

[Filebeat] Allow - for source IP for AWS S3 Access pipeline#26940
kaiyan-sheng merged 2 commits intoelastic:masterfrom
legoguy1000:26913-awss3-update

Conversation

@legoguy1000
Copy link
Copy Markdown
Contributor

@legoguy1000 legoguy1000 commented Jul 16, 2021 •
edited
Loading

What does this PR do?

Allows for a - in place of the remote IP in the AWS S3 Access ingest pipeline.

Why is it important?

When there isn't a value for a specific field in the AWS S3 Access logs, a - is used in its place. Currently if the remote IP is not recorded and a - is present, the grok pattern doesn't match.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=aws TESTING_FILEBEAT_FILESETS=s3access mage -v pythonIntegTest

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 16, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 16, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-16T21:01:37.604+0000

  • Duration: 93 min 2 sec

  • Commit: 20ae326

Test stats 🧪

Test Results
Failed 0
Passed 7872
Skipped 1201
Total 9073

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7872
Skipped 1201
Total 9073

@kaiyan-sheng kaiyan-sheng self-assigned this Jul 16, 2021
@kaiyan-sheng kaiyan-sheng added backport-v7.14.0 Automated backport with mergify backport-v7.15.0 Automated backport with mergify bug needs_integration_sync Changes in this PR need synced to elastic/integrations. Team:Integrations Label for the Integrations team labels Jul 16, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 16, 2021

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 16, 2021
@kaiyan-sheng kaiyan-sheng requested a review from leehinman July 16, 2021 21:01
@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Jul 16, 2021

/test

kaiyan-sheng
kaiyan-sheng approved these changes Jul 16, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this, looks good to me.

@legoguy1000
Copy link
Copy Markdown
Contributor Author

legoguy1000 commented Jul 19, 2021

@kaiyan-sheng looks like it's ready to merge.

leehinman
leehinman approved these changes Jul 19, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaiyan-sheng kaiyan-sheng merged commit f5d33ac into elastic:master Jul 19, 2021
mergify bot pushed a commit that referenced this pull request Jul 19, 2021
@legoguy1000
[Filebeat] Allow - for source IP for AWS S3 Access pipeline (#26940)
435f6a7 
(cherry picked from commit f5d33ac)
@mergify mergify bot mentioned this pull request Jul 19, 2021
Merged
mergify bot pushed a commit that referenced this pull request Jul 19, 2021
@legoguy1000
[Filebeat] Allow - for source IP for AWS S3 Access pipeline (#26940)
6dd877c 
(cherry picked from commit f5d33ac)
@mergify mergify bot mentioned this pull request Jul 19, 2021
Merged
kaiyan-sheng pushed a commit that referenced this pull request Jul 19, 2021
@mergify @legoguy1000
[Filebeat] Allow - for source IP for AWS S3 Access pipeline (#26940) (#…
cda9e92 
…26954)

(cherry picked from commit f5d33ac)

Co-authored-by: Alex Resnick <adr8292@gmail.com>
kaiyan-sheng pushed a commit that referenced this pull request Jul 19, 2021
@mergify @legoguy1000
[Filebeat] Allow - for source IP for AWS S3 Access pipeline (#26940) (#…
b9ccaeb 
…26955)

(cherry picked from commit f5d33ac)

Co-authored-by: Alex Resnick <adr8292@gmail.com>
@kaiyan-sheng kaiyan-sheng mentioned this pull request Jul 19, 2021
Merged
4 tasks
@kaiyan-sheng kaiyan-sheng removed the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jul 19, 2021
@legoguy1000 legoguy1000 deleted the 26913-awss3-update branch July 20, 2021 00:20
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jul 20, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
05ab4ab 
* master:
  Forward port 7.13.4 to master (elastic#26971)
  Use MustAddMetricSet in all metricsets (elastic#26907)
  add_process_metadata: enrich process info with process owner (elastic#21068) (elastic#21111)
  Use aws sdk paginator for FilterLogEvents and GetMetricData (elastic#26852)
  [Filebeat] Allow - for source IP for AWS S3 Access pipeline (elastic#26940)
  Increase timeout to 30secs (elastic#26841)
  Add Cluster filter on Kubernetes Overview ECS dashboard (elastic#26919)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman approved these changes

+1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport-v7.14.0 Automated backport with mergify backport-v7.15.0 Automated backport with mergify bug Team:Integrations Label for the Integrations team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Filebeat AWS Module (S3 Access Log) Grok Pattern Issue

4 participants

@legoguy1000 @elasticmachine @kaiyan-sheng @leehinman