Upgrade AWS SDK v1 to v2 by dblock · Pull Request #7372 · opensearch-project/OpenSearch

dblock · 2023-05-02T21:24:55Z

Description

Upgrading AWS SDK v1 to v2 in discovery-ec2 and repository-s3.

This PR contains the following potentially breaking changes for end-users.

The AWS SDK v2 uses aws.ec2MetadataServiceEndpoint instead of com.amazonaws.sdk.ec2MetadataServiceEndpointOverride. If a user set the IMDS endpoint before that will no longer work without a change. Similarly we've renamed com.amazonaws.sdk.stsEndpointOverride to aws.stsEndpoint. What are the use-cases where anyone would actually do this, or was it done for testing only?
AWS SDK v2 introduced a concept of TokenBucketRetryCondition, which effectively limits retries on a bucket level for APIs that have been failing continuously. It allocates tokens (default is 500), which means that once 500 retries fail for any API on a bucket, new retries will only be allowed once some retries are rejected.

The AWS SDK has other changes, see https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html and https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md for all changes between AWS SDK v1 and v2.

Related Issues

Resolves #7312.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Bukhtawar · 2023-05-03T09:51:42Z

Code scan for dependencies, looks like we have just the two plugins consuming AWS SDK

Signed-off-by: dblock <dblock@amazon.com>

….ec2MetadataServiceEndpoint per https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html. Signed-off-by: dblock <dblock@amazon.com>

Signed-off-by: dblock <dblock@amazon.com>

… as a host. Signed-off-by: dblock <dblock@amazon.com>

Signed-off-by: dblock <dblock@amazon.com>

github-actions · 2023-06-02T11:03:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/16715/
CommitID: 5ceface
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-02T11:23:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/16716/
CommitID: 5988b08
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-02T11:39:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/16717/
CommitID: cfe9809
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>

github-actions · 2023-06-02T12:35:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/16721/
CommitID: a08cb1f

raghuvanshraj · 2023-06-02T14:09:14Z

@Bukhtawar I've captured all follow up items here: #7885

reta · 2023-06-02T15:53:46Z

@dblock we are bringing it to 2.x ? :-)

dblock · 2023-06-02T18:06:50Z

@dblock we are bringing it to 2.x ? :-)

Yes, we should. However we do have some seemingly breaking changes (see description). WDYT? I'll start a backport anyway.

opensearch-trigger-bot · 2023-06-02T18:08:16Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7372-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b15ffc9caaec0303b91ecaff1ab39bb9e860ba48
# Push it to GitHub
git push --set-upstream origin backport/backport-7372-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7372-to-2.x.

reta · 2023-06-02T18:08:49Z

We do, but this is a plugin (and those are long due changes), thanks @dblock !

dblock · 2023-06-02T19:32:24Z

Backport failed above, will need to do a manual one. I can do it next week, @raghuvanshraj lmk if you want to beat me to it.

* AWS SDK v2. * Migrating com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws.ec2MetadataServiceEndpoint per https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html. * Fetch host address for EC2 metadata service from SdkSettings. * Deprecations of running with partial keys were removed in AWS SDK v2. * Underscore is not a valid host name, thus URI.create doesn't treat it as a host. * Added missing JARs, all tests pass. * Setup a default AWS profile under config, don't read ~/.aws. * Re-enable availability zone node attribute implementation. * Cleaning up privileged calls using SocketAccess * Consistently setup and teardown AWS settings in repository-s3. * Using SdkException instead of RuntimeException in S3BlobContainer * Removing AwsSessionCredentials object from S3BasicSesstionCredentials * Removing S3BasicCredentials and S3BasicSessionCredentials Signed-off-by: dblock <dblock@amazon.com> Co-authored-by: Raghuvansh Raj <raghraaj@amazon.com> (cherry picked from commit b15ffc9)

…t#7926) * AWS SDK v2. * Migrating com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws.ec2MetadataServiceEndpoint per https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html. * Fetch host address for EC2 metadata service from SdkSettings. * Deprecations of running with partial keys were removed in AWS SDK v2. * Underscore is not a valid host name, thus URI.create doesn't treat it as a host. * Added missing JARs, all tests pass. * Setup a default AWS profile under config, don't read ~/.aws. * Re-enable availability zone node attribute implementation. * Cleaning up privileged calls using SocketAccess * Consistently setup and teardown AWS settings in repository-s3. * Using SdkException instead of RuntimeException in S3BlobContainer * Removing AwsSessionCredentials object from S3BasicSesstionCredentials * Removing S3BasicCredentials and S3BasicSessionCredentials Signed-off-by: dblock <dblock@amazon.com> Co-authored-by: Raghuvansh Raj <raghraaj@amazon.com> (cherry picked from commit b15ffc9)

* AWS SDK v2. * Migrating com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws.ec2MetadataServiceEndpoint per https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html. * Fetch host address for EC2 metadata service from SdkSettings. * Deprecations of running with partial keys were removed in AWS SDK v2. * Underscore is not a valid host name, thus URI.create doesn't treat it as a host. * Added missing JARs, all tests pass. * Setup a default AWS profile under config, don't read ~/.aws. * Re-enable availability zone node attribute implementation. * Cleaning up privileged calls using SocketAccess * Consistently setup and teardown AWS settings in repository-s3. * Using SdkException instead of RuntimeException in S3BlobContainer * Removing AwsSessionCredentials object from S3BasicSesstionCredentials * Removing S3BasicCredentials and S3BasicSessionCredentials Signed-off-by: dblock <dblock@amazon.com> Co-authored-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

dblock force-pushed the aws-sdk-v2 branch from b2e2d47 to fdf8453 Compare May 2, 2023 21:26

This comment was marked as outdated.

Sign in to view

dblock mentioned this pull request May 2, 2023

Upgrade AWS SDK to v2 #7312

Closed

2 tasks

This comment was marked as outdated.

Sign in to view

dblock mentioned this pull request May 3, 2023

[Remote Store] Add netty dependencies in repository-s3 plugin #7216

Merged

6 tasks

dblock changed the title ~~Upgrade AWS SDK v1 to v2~~ Upgrade AWS SDK v1 to v2 in discovery-ec2 May 3, 2023

dblock force-pushed the aws-sdk-v2 branch from fdf8453 to 2b3980f Compare May 8, 2023 15:26

This comment was marked as outdated.

Sign in to view

dblock force-pushed the aws-sdk-v2 branch from 2b3980f to 29f2369 Compare May 9, 2023 15:30

This comment was marked as outdated.

Sign in to view

dblock added 9 commits May 15, 2023 07:51

WIP: AWS SDK v2.

457c199

Signed-off-by: dblock <dblock@amazon.com>

Migrating com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws…

50b2a19

….ec2MetadataServiceEndpoint per https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html. Signed-off-by: dblock <dblock@amazon.com>

Fetch host address for EC2 metadata service from SdkSettings.

dd460e8

Signed-off-by: dblock <dblock@amazon.com>

Added missing runtime classes.

4f62701

Signed-off-by: dblock <dblock@amazon.com>

Deprecations of running with partial keys were removed in AWS SDK v2.

5848a82

Signed-off-by: dblock <dblock@amazon.com>

Underscore is not a valid host name, thus URI.create doesn't treat it…

a896310

… as a host. Signed-off-by: dblock <dblock@amazon.com>

Added missing JARs, all tests pass.

3a13b2e

Signed-off-by: dblock <dblock@amazon.com>

Setup a default AWS profile under config, don't read ~/.aws.

033af9d

Signed-off-by: dblock <dblock@amazon.com>

Revert changes in repository-s3.

492215a

Signed-off-by: dblock <dblock@amazon.com>

dblock force-pushed the aws-sdk-v2 branch from b182eed to 492215a Compare May 15, 2023 11:52

This comment was marked as outdated.

Sign in to view

dblock added 2 commits May 15, 2023 10:24

Re-enable availability zone node attribute implementation.

8882143

Signed-off-by: dblock <dblock@amazon.com>

Implement missing TODOs in UTs.

d79e658

Signed-off-by: dblock <dblock@amazon.com>

This comment was marked as outdated.

Sign in to view

Re-implemented proxy configuration tests.

f6b3a4c

Signed-off-by: dblock <dblock@amazon.com>

This comment was marked as outdated.

Sign in to view

Write test java.policy.

0d501b3

Signed-off-by: dblock <dblock@amazon.com>

This comment was marked as outdated.

Sign in to view

Bukhtawar reviewed Jun 2, 2023

View reviewed changes

Comment thread plugins/discovery-ec2/src/main/java/org/opensearch/discovery/ec2/Ec2DiscoveryPlugin.java

raghuvanshraj force-pushed the aws-sdk-v2 branch from 5ceface to 5988b08 Compare June 2, 2023 11:14

raghuvanshraj force-pushed the aws-sdk-v2 branch from 5988b08 to cfe9809 Compare June 2, 2023 11:28

Adding UTs for S3BlobContainer

a08cb1f

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>

raghuvanshraj force-pushed the aws-sdk-v2 branch from cfe9809 to a08cb1f Compare June 2, 2023 11:58

Bukhtawar approved these changes Jun 2, 2023

View reviewed changes

Bukhtawar merged commit b15ffc9 into opensearch-project:main Jun 2, 2023

dblock deleted the aws-sdk-v2 branch June 2, 2023 18:07

dblock added the backport 2.x Backport to 2.x branch label Jun 2, 2023

dblock mentioned this pull request Jun 5, 2023

Use IMDSv2 in disvovery-ec2 #7916

Closed

dblock mentioned this pull request Jun 6, 2023

[Backport 2.x] Backporting #7372 to 2.x (upgrade to AWS SDK v2) #7926

Merged

6 tasks

This was referenced Jun 6, 2023

Upgrade EC2 discovery plugin to use IMDSv2 #3040

Open

Add amazon_s3_region to gradle check CI/CD opensearch-project/opensearch-build#3615

Open

dblock mentioned this pull request Jul 19, 2023

Upgrade AWS SDK used by the repository-s3 plugin to allow use of IMDSv2 #3690

Closed

nknize mentioned this pull request Oct 26, 2023

[Remove] Remaining Joda and Joda Dependency #10949

Closed

rtsui-harmonicinc mentioned this pull request Jul 18, 2025

[BUG][Opensearch] Opensearch snapshot management: can't select AWS region for configuring s3 bucket. #9265

Closed

Conversation

dblock commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Bukhtawar commented May 3, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

github-actions Bot commented Jun 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

raghuvanshraj commented Jun 2, 2023

Uh oh!

reta commented Jun 2, 2023

Uh oh!

dblock commented Jun 2, 2023

Uh oh!

opensearch-trigger-bot Bot commented Jun 2, 2023

Uh oh!

reta commented Jun 2, 2023

Uh oh!

dblock commented Jun 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dblock commented May 2, 2023 •

edited

Loading