[Feature Branch] Upgrade AWS SDK v2 library to the latest version by sayden · Pull Request #31224 · elastic/beats

sayden · 2022-04-07T19:39:15Z

What does this PR do?

This PR upgrades the current AWS SDK library we use to the latest version. It should include all the latest fixes from AWS as well as many corrections in our code to keep working with it.

Consider this a feature branch where smaller areas of our code will be gradually integrated, like metricsets and inputs on different PR's.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Upgrade the aws-sdk-go-v2 version in beats #26873

Status

Added changes required in Cloudwatch Input

Break down tasks for testing this PR

Metricbeat CloudWatch metricset to make sure metrics are collected correctly with Access keys authentication - @kaiyan-sheng
Metricbeat CloudWatch metricset to make sure metrics are collected correctly with credential profile name authentication - @girodav
Metricbeat CloudWatch metricset to make sure metrics are collected correctly with role arn authentication @kaiyan-sheng
Metricbeat EC2 metricset to make sure metadata is collected for EC2 - @girodav
Filebeat s3 input with SQS - @aspacca run the integration tests locally
Filebeat s3 input without SQS - @aspacca run the integration tests locally
Filebeat cloudwatch input @sayden (@aspacca run the integration tests locally)

…g feature that has been deleted.

…m is not obvious to work package by package

elasticmachine · 2022-04-07T19:47:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-22T11:31:15.133+0000
Duration: 201 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	24418
Skipped	2106
Total	26524

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…g feature that has been deleted.

…m is not obvious to work package by package

mergify · 2022-07-06T22:01:45Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

francescayeye · 2022-07-07T03:25:45Z

@kaiyan-sheng the PR seems almost complete

I'd like to review the validity my initial comments: #31224 (comment) and #31224 (comment)

would you mind checking them?

thanks

…pgrade-library

francescayeye

here how I implemented pagination error handling on the s3 objects listing:
https://github.com/elastic/beats/pull/31224/files#diff-c6357a5d68a965123a3b3dc3cbf0bddfe7a5628dceb5327885da33f1735a3facR158-R173

it might be too defensive, up to you if using the same pattern for the two paginators in this commit

in the case of s3 ListObjectsPaginator.NextPage() an error can be caused by !ListObjectsPaginator.HasMorePages(). we should never reach such state since the for loop would have already exited.

if the cwlogs LogEventsPaginator.NextPage() and DescribeLogGroupsPaginator.NextPage() do not have an error caused by !HasMorePages() no need of the extra if

mergify · 2022-07-11T02:01:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

mergify · 2022-07-11T13:40:23Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

kaiyan-sheng · 2022-07-11T20:10:35Z

in the case of s3 ListObjectsPaginator.NextPage() an error can be caused by !ListObjectsPaginator.HasMorePages(). we should never reach such state since the for loop would have already exited.

if the cwlogs LogEventsPaginator.NextPage() and DescribeLogGroupsPaginator.NextPage() do not have an error caused by !HasMorePages() no need of the extra if

I checked the aws-sdk-go-v2, both FilterLogEventsPaginator.NextPage() and DescribeLogGroupsPaginator.NextPage() also check for error caused by !HasMorePages(). But I don't think we absolutely require the additional check for !paginator.HasMorePages() since that would never be true with the condition of the for loop right above. Do you think it's required?

…pgrade-library

mergify · 2022-07-13T15:50:19Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

…pgrade-library

mergify · 2022-07-20T21:51:17Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

…pgrade-library

…nput

mergify · 2022-07-21T15:48:40Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/xpack/aws/upgrade-library upstream/feature/xpack/aws/upgrade-library
git merge upstream/main
git push upstream feature/xpack/aws/upgrade-library

…pgrade-library

francescayeye · 2022-08-23T07:37:50Z

/test

…1224)

sayden added 4 commits April 2, 2022 21:28

Upgrade Cloudwatch library to avoid the use of CloudwatchIface mockin…

1baf5c8

…g feature that has been deleted.

Removed unused function from cloudwatch input tests

0be4cb3

Add all required missing libraries because the dependency between the…

d84d46d

…m is not obvious to work package by package

A small thing that was missing in the test of cloudwatch input

ef53a61

sayden added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Apr 7, 2022

sayden requested a review from kaiyan-sheng April 7, 2022 19:39

sayden self-assigned this Apr 7, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 7, 2022

Fix some lefovers

6952ce1

sayden mentioned this pull request Apr 7, 2022

[Feature Branch] Upgrade AWS SDK v2 library to the latest version #31208

Closed

6 tasks

sayden marked this pull request as ready for review April 13, 2022 07:03

sayden requested a review from a team as a code owner April 13, 2022 07:03

sayden mentioned this pull request Apr 13, 2022

[AWS SDK v2] Upgrade library common credentials #31225

Closed

cmacknz removed the request for review from a team April 13, 2022 18:54

sayden added 14 commits April 14, 2022 19:55

Upgrade Cloudwatch library to avoid the use of CloudwatchIface mockin…

0713e0b

…g feature that has been deleted.

Removed unused function from cloudwatch input tests

7c505b1

Add all required missing libraries because the dependency between the…

22d202c

…m is not obvious to work package by package

A small thing that was missing in the test of cloudwatch input

6a66ae7

Add the modifications required for the common credentials.go file

f8a5d7c

fix test file for credentials

8eb4153

Add the changes on one last PR

61d6e9b

Go Mod changes

2b4d084

some integration test for cloudwatch input that was missing

ab5c547

s3 input code only (no tests)

8553561

EC2 code only (no tests)

4a6fd74

EC2 code only (no tests) (now for real)

4012ef6

ELB Autodiscover code only (no tests)

8dc819a

Credentials code only (no tests)

bb2e0dd

kaiyan-sheng added 4 commits July 8, 2022 14:16

use paginator for DescribeLogGroups and FilterLogEvents

6501dba

Merge remote-tracking branch 'upstream/main' into feature/xpack/aws/u…

e04cbe8

…pgrade-library

update notice.txt

2077905

address comment

a5672a5

francescayeye reviewed Jul 11, 2022

View reviewed changes

fix merge main

210be0a

Merge remote-tracking branch 'upstream/main' into feature/xpack/aws/u…

8dd98e6

…pgrade-library

Merge remote-tracking branch 'upstream/main' into feature/xpack/aws/u…

be2c53b

…pgrade-library

Andrea Spacca added 2 commits July 21, 2022 21:17

Merge remote-tracking branch 'upstream/main' into feature/xpack/aws/u…

1b6e7b6

…pgrade-library

fixes for integration tests, remove unused store from awscloudwatch i…

6d7ff4b

…nput

kaiyan-sheng and others added 4 commits July 21, 2022 12:25

Merge remote-tracking branch 'upstream/main' into feature/xpack/aws/u…

267c009

…pgrade-library

fix error check in cloudwatch input

4bb4a54

run go integ test on cloudAWS

00cef6e

abs path for terraform outputfile

32898d9

kaiyan-sheng approved these changes Jul 22, 2022

View reviewed changes

sayden merged commit f5ff0a9 into main Jul 22, 2022

kaiyan-sheng deleted the feature/xpack/aws/upgrade-library branch July 22, 2022 15:44

francescayeye mentioned this pull request Jul 26, 2022

Build 705 for main with status FAILURE #32473

Closed

gpop63 mentioned this pull request Jul 27, 2022

error handling for cloudwatch on getLogEventsFromCloudWatch #32436

Closed

tommyers-elastic mentioned this pull request Aug 26, 2022

[filebeat] Credentials error with S3+SQS input in 8.4.0 #32888

Closed

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

[Feature Branch] Upgrade AWS SDK v2 library to the latest version (#3…

455cee5

…1224)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature Branch] Upgrade AWS SDK v2 library to the latest version#31224

[Feature Branch] Upgrade AWS SDK v2 library to the latest version#31224
sayden merged 82 commits intomainfrom
feature/xpack/aws/upgrade-library

sayden commented Apr 7, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Apr 7, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

mergify bot commented Jul 6, 2022

Uh oh!

francescayeye commented Jul 7, 2022

Uh oh!

francescayeye left a comment

Uh oh!

mergify bot commented Jul 11, 2022

Uh oh!

mergify bot commented Jul 11, 2022

Uh oh!

kaiyan-sheng commented Jul 11, 2022 •

edited

Loading

Uh oh!

mergify bot commented Jul 13, 2022

Uh oh!

mergify bot commented Jul 20, 2022

Uh oh!

mergify bot commented Jul 21, 2022

Uh oh!

francescayeye commented Aug 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sayden commented Apr 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Related issues

Status

Break down tasks for testing this PR

Uh oh!

elasticmachine commented Apr 7, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

mergify bot commented Jul 6, 2022

Uh oh!

francescayeye commented Jul 7, 2022

Uh oh!

francescayeye left a comment

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jul 11, 2022

Uh oh!

mergify bot commented Jul 11, 2022

Uh oh!

kaiyan-sheng commented Jul 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Jul 13, 2022

Uh oh!

mergify bot commented Jul 20, 2022

Uh oh!

mergify bot commented Jul 21, 2022

Uh oh!

francescayeye commented Aug 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sayden commented Apr 7, 2022 •

edited

Loading

elasticmachine commented Apr 7, 2022 •

edited by jenkins-beats-ci bot

Loading

kaiyan-sheng commented Jul 11, 2022 •

edited

Loading