Skip to content

x-pack/filebeat/input/awss3 Add AWS Endpoint resolver#36208

Merged
bhapas merged 18 commits intoelastic:mainfrom
bhapas:fix-enpoint-resolver
Aug 14, 2023
Merged

x-pack/filebeat/input/awss3 Add AWS Endpoint resolver#36208
bhapas merged 18 commits intoelastic:mainfrom
bhapas:fix-enpoint-resolver

Conversation

@bhapas
Copy link
Copy Markdown
Contributor

@bhapas bhapas commented Aug 2, 2023 •
edited
Loading

What does this PR do?

This PR adds endpoint resolver to the AWS config.

Why is it important?

By adding custom endpoint resolver config options to AWSConfig , It becomes flexible to use the input against any AWS like resources, for example , when testing against Localstack.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.
  • I have made my commit title and message explanatory about the purpose and the reason of the change

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 2, 2023
@mergify mergify bot assigned bhapas Aug 2, 2023
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Aug 2, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @bhapas? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@bhapas bhapas changed the title Add AWS Endpoint resolver x-pack/libbeat/common/aws Add AWS Endpoint resolver Aug 2, 2023
@bhapas bhapas force-pushed the fix-enpoint-resolver branch from 931bbea to 65338b8 Compare August 2, 2023 12:52
@bhapas bhapas added Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Aug 2, 2023
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 2, 2023
@bhapas bhapas added needs_team Indicates that the issue/PR needs a Team:* label Team:Security-External Integrations labels Aug 2, 2023
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 2, 2023
@bhapas bhapas added needs_team Indicates that the issue/PR needs a Team:* label backport-skip Skip notification from the automated backport with mergify labels Aug 2, 2023
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Aug 2, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-enpoint-resolver upstream/fix-enpoint-resolver
git merge upstream/main
git push upstream fix-enpoint-resolver

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 2, 2023
@bhapas bhapas force-pushed the fix-enpoint-resolver branch from baad3a6 to 94e159b Compare August 2, 2023 12:55
@bhapas bhapas added 8.10-candidate and removed Team:Elastic-Agent Label for the Agent team labels Aug 2, 2023
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 2, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-08-14T12:00:10.827+0000

  • Duration: 139 min 6 sec

Test stats 🧪

Test Results
Failed 0
Passed 6264
Skipped 353
Total 6617

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@bhapas bhapas force-pushed the fix-enpoint-resolver branch from b5dfa9f to 5b37d6f Compare August 7, 2023 09:11
@bhapas bhapas added backport-v8.9.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Aug 7, 2023
@bhapas
Copy link
Copy Markdown
Contributor Author

bhapas commented Aug 7, 2023

/test

@bhapas bhapas marked this pull request as ready for review August 7, 2023 13:52
@bhapas bhapas requested a review from a team as a code owner August 11, 2023 12:05
@bhapas bhapas requested review from belimawr and rdner August 11, 2023 12:05
@botelastic botelastic bot added the Team:Automation Label for the Observability productivity team label Aug 11, 2023
@bhapas bhapas force-pushed the fix-enpoint-resolver branch from 779b47c to 7373cd3 Compare August 11, 2023 12:49
@bhapas bhapas force-pushed the fix-enpoint-resolver branch from 7373cd3 to ae1b6db Compare August 11, 2023 17:23
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Aug 13, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-enpoint-resolver upstream/fix-enpoint-resolver
git merge upstream/main
git push upstream fix-enpoint-resolver

@bhapas bhapas force-pushed the fix-enpoint-resolver branch from bc82cdc to 43e629a Compare August 14, 2023 09:20
@bhapas bhapas requested review from kaiyan-sheng and removed request for a team, belimawr and rdner August 14, 2023 09:20
@bhapas
Copy link
Copy Markdown
Contributor Author

bhapas commented Aug 14, 2023

/test

@bhapas bhapas force-pushed the fix-enpoint-resolver branch from 1f51ff6 to 5fa2af2 Compare August 14, 2023 11:50
zmoog
zmoog approved these changes Aug 14, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@zmoog zmoog left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

It seems we're using endpoint resolution v1, please correct me if I'm wrong. This is fine if this is the case since AWS recently released AWS will release v2 in Q3.

Are there considerations or plans for v1 vs. v2?

@bhapas
Copy link
Copy Markdown
Contributor Author

bhapas commented Aug 14, 2023 •
edited
Loading

LGTM.

It seems we're using endpoint resolution v1, please correct me if I'm wrong.

This is fine if this is the case since AWS released v2 recently. Are there considerations or plans for v1 vs. v2?

V2 sounds more practical for testing purposes , specially Localstack . Probably would revisit this when the V2 is officially out in Q3 2023

Created an issue to track this - #36319

@bhapas bhapas merged commit 4586146 into elastic:main Aug 14, 2023
@bhapas bhapas deleted the fix-enpoint-resolver branch August 14, 2023 15:55
mergify bot pushed a commit that referenced this pull request Aug 14, 2023
@bhapas @mergify
x-pack/filebeat/input/awss3 Add AWS Endpoint resolver (#36208)
f651e2d 
* Add AWS Endpoint resolver

* Add Signing region to resolver

(cherry picked from commit 4586146)
@mergify mergify bot mentioned this pull request Aug 14, 2023
Merged
@bhapas bhapas mentioned this pull request Jul 31, 2023
Closed
7 tasks
bhapas added a commit that referenced this pull request Aug 14, 2023
@mergify @bhapas
[8.9](backport #36208) x-pack/filebeat/input/awss3 Add AWS Endpoint r…
edea2f3 
…esolver (#36318)

* x-pack/filebeat/input/awss3 Add AWS Endpoint resolver (#36208)

* Add AWS Endpoint resolver

* Add Signing region to resolver

(cherry picked from commit 4586146)

* Update CHANGELOG.next.asciidoc

---------

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@bhapas @Scholar-Li
x-pack/filebeat/input/awss3 Add AWS Endpoint resolver (elastic#36208)
85429a7 
* Add AWS Endpoint resolver

* Add Signing region to resolver
@strawgate strawgate mentioned this pull request May 23, 2024
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zmoog zmoog zmoog approved these changes

@kaiyan-sheng kaiyan-sheng Awaiting requested review from kaiyan-sheng

Assignees

@bhapas bhapas

Labels

8.10-candidate backport-v8.9.0 Automated backport with mergify Team:Automation Label for the Observability productivity team Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bhapas @elasticmachine @zmoog @kaiyan-sheng