[Filebeat] Fix hardcoded amazonaws.com endpoint by legoguy1000 · Pull Request #24861 · elastic/beats

legoguy1000 · 2021-03-30T23:49:51Z

What does this PR do?

Updates the S3 input validation for the queue url to check for the amazonaws.com domain or for the domain to match the endpoint config parameter that already exists.

Why is it important?

Adds the ability for to use non amazonaws.com domains for Filebeat S3 input.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Related issues

closes #24420

Use cases

Screenshots

Logs

legoguy1000 · 2021-03-30T23:58:49Z

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

elasticmachine · 2021-03-31T00:12:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: kaiyan-sheng commented: /test
Start Time: 2021-04-08T16:11:55.980+0000
Duration: 50 min 2 sec
Commit: 85d5b0f

Test stats 🧪

Test	Results
Failed	0
Passed	6910
Skipped	1185
Total	8095

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6910
Skipped	1185
Total	8095

elasticmachine · 2021-03-31T07:00:13Z

Pinging @elastic/integrations (Team:Integrations)

botelastic · 2021-03-31T07:00:21Z

This pull request doesn't have a Team:<team> label.

kaiyan-sheng · 2021-03-31T07:04:04Z

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

@legoguy1000 Thanks for your contribution! We have unit tests for the golang code and for this function getRegionFromQueueURL specifically, there is TestGetRegionFromQueueURL under beats/x-pack/filebeat/input/awss3/collector_test.go. Please add some test cases there for URLs that have a different endpoint. Thank you!!

x-pack/filebeat/input/awss3/collector.go

legoguy1000 · 2021-03-31T13:30:52Z

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

@legoguy1000 Thanks for your contribution! We have unit tests for the golang code and for this function getRegionFromQueueURL specifically, there is TestGetRegionFromQueueURL under beats/x-pack/filebeat/input/awss3/collector_test.go. Please add some test cases there for URLs that have a different endpoint. Thank you!!

Done.

x-pack/filebeat/input/awss3/collector_test.go

x-pack/filebeat/input/awss3/collector.go

kaiyan-sheng · 2021-04-08T16:11:35Z

/test

kaiyan-sheng

Thanks for your contribution!! This PR looks good to me!

(cherry picked from commit 7b729da) Co-authored-by: Alex Resnick <adr8292@gmail.com>

* upstream/master: (308 commits) [winlogbeat] Add support for sysmon v13 events 24 and 25 (elastic#24945) mergify: add backport label (elastic#25050) Add pod.ip in k8s metadata (elastic#25037) [elastic-agent] Use fleet.url for container cmd (elastic#25026) disable TestXPackEnabled flaky test in logstash metricbeat module (elastic#25034) Leverege leader election in agent k8s manifests (elastic#25016) libbeat/publisher/pipeline: expand monitoring (elastic#24700) libbeat: fix decode_json_fields config validation (elastic#24862) Remove make docs-preview instructions (elastic#25001) [Filebeat] Fix IPtables pipeline (elastic#24928) [DOCS] cd into correct directory before invoking mage. (elastic#17679) Add -buildmode=pie for supported platform (elastic#24964) Add agent's direcotry in k8s manifest generator (elastic#24987) [mergify] assign the original author (elastic#25007) Fix AWS module flaky tests (elastic#24852) [filebeat] Use fail_on_template_error on google_workspace and okta pagination (elastic#24967) Updated config to match defaults (elastic#25004) [Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861) Add cloud.service.name to add_cloud_metadata (elastic#24993) [Ingest Manager] Expose processes and their metrics (elastic#24788) ...

…tic#25010) (cherry picked from commit 7129c21) Co-authored-by: Alex Resnick <adr8292@gmail.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 30, 2021

legoguy1000 force-pushed the 24420-s3-endpoint branch from 0de9ed1 to 563cfbf Compare March 30, 2021 23:56

elastic#24420: Fix hardcoded amazonaws.com endpoint

8c8f721

legoguy1000 force-pushed the 24420-s3-endpoint branch from 563cfbf to 8c8f721 Compare March 30, 2021 23:57

legoguy1000 marked this pull request as ready for review March 30, 2021 23:57

kaiyan-sheng self-assigned this Mar 31, 2021

kaiyan-sheng added the Team:Integrations Label for the Integrations team label Mar 31, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2021

kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. needs_team Indicates that the issue/PR needs a Team:* label labels Mar 31, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2021

kaiyan-sheng added the test-plan Add this PR to be manual test plan label Mar 31, 2021

kaiyan-sheng reviewed Mar 31, 2021

View reviewed changes

x-pack/filebeat/input/awss3/collector.go Outdated Show resolved Hide resolved

Update unit tests

d375c63

kaiyan-sheng reviewed Apr 1, 2021

View reviewed changes

x-pack/filebeat/input/awss3/collector_test.go Outdated Show resolved Hide resolved

Update tests per review

5569cf0

legoguy1000 requested a review from kaiyan-sheng April 2, 2021 11:50

kaiyan-sheng reviewed Apr 6, 2021

View reviewed changes

x-pack/filebeat/input/awss3/collector.go Outdated Show resolved Hide resolved

update error output and add test

85d5b0f

legoguy1000 requested a review from kaiyan-sheng April 7, 2021 13:29

kaiyan-sheng approved these changes Apr 9, 2021

View reviewed changes

kaiyan-sheng merged commit 7b729da into elastic:master Apr 9, 2021

kaiyan-sheng removed the test-plan Add this PR to be manual test plan label Apr 9, 2021

kaiyan-sheng mentioned this pull request Apr 9, 2021

Cherry-pick #24861 to 7.x: [Filebeat] Fix hardcoded amazonaws.com endpoint #25009

Merged

6 tasks

kaiyan-sheng added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 9, 2021

kaiyan-sheng mentioned this pull request Apr 9, 2021

Cherry-pick #24861 to 7.12: [Filebeat] Fix hardcoded amazonaws.com endpoint #25010

Merged

6 tasks

kaiyan-sheng added the v7.12.1 label Apr 9, 2021

kaiyan-sheng pushed a commit that referenced this pull request Apr 10, 2021

[Filebeat] Fix hardcoded amazonaws.com endpoint (#24861) (#25009)

7428953

(cherry picked from commit 7b729da) Co-authored-by: Alex Resnick <adr8292@gmail.com>

kaiyan-sheng pushed a commit that referenced this pull request Apr 10, 2021

[Filebeat] Fix hardcoded amazonaws.com endpoint (#24861) (#25010)

34a84af

(cherry picked from commit 7b729da) Co-authored-by: Alex Resnick <adr8292@gmail.com>

legoguy1000 deleted the 24420-s3-endpoint branch April 29, 2021 00:39

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

[Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861) (elas…

cb71daf

…tic#25010) (cherry picked from commit 7129c21) Co-authored-by: Alex Resnick <adr8292@gmail.com>

strawgate mentioned this pull request May 24, 2024

Cannot use a custom Endpoint with SQS #39706

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Fix hardcoded amazonaws.com endpoint#24861

[Filebeat] Fix hardcoded amazonaws.com endpoint#24861
kaiyan-sheng merged 4 commits intoelastic:masterfrom
legoguy1000:24420-s3-endpoint

legoguy1000 commented Mar 30, 2021 •

edited

Loading

Uh oh!

legoguy1000 commented Mar 30, 2021

Uh oh!

elasticmachine commented Mar 31, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

elasticmachine commented Mar 31, 2021

Uh oh!

botelastic bot commented Mar 31, 2021

Uh oh!

kaiyan-sheng commented Mar 31, 2021

Uh oh!

Uh oh!

legoguy1000 commented Mar 31, 2021

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng commented Apr 8, 2021

Uh oh!

kaiyan-sheng left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

legoguy1000 commented Mar 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

legoguy1000 commented Mar 30, 2021

Uh oh!

elasticmachine commented Mar 31, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Mar 31, 2021

Uh oh!

botelastic bot commented Mar 31, 2021

Uh oh!

kaiyan-sheng commented Mar 31, 2021

Uh oh!

Uh oh!

legoguy1000 commented Mar 31, 2021

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng commented Apr 8, 2021

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

legoguy1000 commented Mar 30, 2021 •

edited

Loading

elasticmachine commented Mar 31, 2021 •

edited by jenkins-beats-ci bot

Loading