Skip to content

Set up AWS SDK v2 and migrate STS service#4926

Merged
cPu1 merged 9 commits intoeksctl-io:mainfrom
cPu1:aws-sdk-v2
Mar 23, 2022
Merged

Set up AWS SDK v2 and migrate STS service#4926
cPu1 merged 9 commits intoeksctl-io:mainfrom
cPu1:aws-sdk-v2

Conversation

@cPu1
Copy link
Copy Markdown
Contributor

@cPu1 cPu1 commented Mar 10, 2022

Description

This changelist configures aws-sdk-go-v2 and adds interfaces and types for obtaining AWS services. A code generation script is added that, as part of the build process, generates interfaces from AWS SDK types for mocking. Due to this issue in ifacemaker, it passes the source generated by ifacemaker through another script add_import.go that adds the missing import.

As part of #4877, the STS service is migrated to use aws-sdk-go-v2. One usage of STS cannot be migrated to v2 yet as it's blocked on sigs.k8s.io/aws-iam-authenticator/pkg/token.

Closes #4925
Closes #4877

TODO:

Checklist

  • Added tests that cover your change (if possible)
  • Added/modified documentation as required (such as the README.md, or the userdocs directory)
  • Manually tested
  • Made sure the title of the PR is a good description that can go into the release notes
  • (Core team) Added labels for change area (e.g. area/nodegroup) and kind (e.g. kind/improvement)

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

  • Backfilled missing tests for code in same general area 🎉
  • Refactored something and made the world a better place 🌟

@cPu1 cPu1 added area/tech-debt Leftover improvements in code, testing and building dependencies Pull requests that update a dependency file labels Mar 10, 2022
@cPu1 cPu1 force-pushed the aws-sdk-v2 branch 3 times, most recently from b6186d4 to 491270a Compare March 11, 2022 09:28
@cPu1 cPu1 marked this pull request as ready for review March 11, 2022 10:41
@cPu1 cPu1 added kind/feature New feature or request kind/improvement and removed kind/feature New feature or request labels Mar 11, 2022
Comment on lines +21 to +76
func addDotImport(code []byte, packageName string) error {
d := decorator.NewDecorator(token.NewFileSet())
f, err := d.Parse(code)
if err != nil {
return err
}

for _, dl := range f.Decls {
gd, ok := dl.(*dst.GenDecl)
if !ok {
logger.Warning("expected type %T; got %T", &dst.GenDecl{}, dl)
continue
}

if gd.Tok == token.IMPORT {
gd.Specs = append(gd.Specs, &dst.ImportSpec{
Name: &dst.Ident{
Name: ".",
},
Path: &dst.BasicLit{Kind: token.STRING, Value: strconv.Quote(packageName)},
})
}
}

restorer := decorator.NewRestorer()
var buf bytes.Buffer
if err := restorer.Fprint(&buf, f); err != nil {
return fmt.Errorf("error restoring source: %w", err)
}
src, err := format.Source(buf.Bytes())
if err != nil {
return fmt.Errorf("error formatting source: %w", src)
}
if _, err := os.Stdout.Write(src); err != nil {
return fmt.Errorf("error writing source to STDOUT: %w", err)
}
return nil
}

func main() {
if len(os.Args) != 2 {
panic(fmt.Sprintf("usage: add_import <package-to-import>"))
}
source, err := ioutil.ReadAll(os.Stdin)
if err != nil {
panic(fmt.Errorf("unexpected error reading from STDIN: %w", err))
}

packageName := os.Args[1]
if err = addDotImport(source, packageName); err != nil {
panic(err)
}
}
Copy link
Copy Markdown
Contributor

@aclevername aclevername Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these 73 lines of code really easier than just having the 1 sed 🤔 ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's by no means easier than sed, but it's more robust IMO. We can also modify this program later to avoid using dot imports by qualifying local types with the package name.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also modify this program later to avoid using dot imports by qualifying local types with the package name.

do we need to do that though 🤷 ?

It's by no means easier than sed, but it's more robust IMO.

I would disagree, the contents of addImport is pretty obscure, on a first pass I struggle to understand what its disagree. The sed is just a bit simpler to read, and much easier to test manually

Copy link
Copy Markdown
Contributor

@nikimanoledaki nikimanoledaki Mar 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's more robust IMO

I'm trying to understand the pros/cons of each method, sed vs addDotImport. @cPu1 could you expand a bit on this, please? addDotImport adds quite a lot of complexity compared to sed. What do you see as the issues or limitations of sed, and how does addDotImport solve this and perform better?

If we do go with this program, I would suggest adding at least the minimum amount of unit tests to ensure that it really is more robust, that it does what we expect it to do, and to make it more maintainable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nikimanoledaki sed is not the right tool to manipulate Go sources. That sed snippet can generate programs that have syntax errors.

Here's what's wrong with using that sed snippet:

$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
// Code generated by ifacemaker; DO NOT EDIT.

package awsapi

import "context"

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}

EOF

// Code generated by ifacemaker; DO NOT EDIT.

package awsapi

import "context"
. "github.com/aws/aws-sdk-go-v2/service/eks"

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}


$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
// Code generated by ifacemaker; DO NOT EDIT.
// Copyright 2020 Weaveworks.
// The awsapi package provides an interface to AWS services. The "context" package is used to carry deadlines and
// cancellation signals.

package awsapi

import (
    "context"
)

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}

EOF

// Code generated by ifacemaker; DO NOT EDIT.
// Copyright 2020 Weaveworks.
// The awsapi package provides an interface to AWS services. The "context" package is used to carry deadlines and
. "github.com/aws/aws-sdk-go-v2/service/eks"
// cancellation signals.

package awsapi

import (
    "context"
. "github.com/aws/aws-sdk-go-v2/service/eks"
)

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}


$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
package awsapi

import (
    "fmt"
)

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}

EOF

package awsapi

import (
    "fmt"
)

// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
 // ...
}

As for the maintenance of the add_import.go program, we have an issue for that: #4994

@cPu1 cPu1 force-pushed the aws-sdk-v2 branch 4 times, most recently from 327b3c5 to 676ae71 Compare March 11, 2022 13:56
@cPu1 cPu1 mentioned this pull request Mar 11, 2022
7 tasks
@aclevername aclevername requested a review from a team March 11, 2022 15:58
Comment on lines +12 to +58
// ServicesV2 implements api.ServicesV2
type ServicesV2 struct {
config aws.Config
}

// STSV2 implements the AWS STS service
func (s *ServicesV2) STSV2() awsapi.STS {
return sts.NewFromConfig(s.config, func(o *sts.Options) {
// Disable retryer for STS
// (see https://github.com/weaveworks/eksctl/issues/705)
o.Retryer = aws.NopRetryer{}
})
}

// CloudFormationV2 implements the AWS CloudFormation service
func (s *ServicesV2) CloudFormationV2() awsapi.CloudFormation {
return cloudformation.NewFromConfig(s.config, func(o *cloudformation.Options) {
// Use adaptive mode for retrying CloudFormation requests to mimic
// the logic used for AWS SDK v1.
o.Retryer = retry.NewAdaptiveMode(func(o *retry.AdaptiveModeOptions) {
o.StandardOptions = []func(*retry.StandardOptions){
func(so *retry.StandardOptions) {
so.MaxAttempts = maxRetries
},
}
})
})
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be removed right?

Copy link
Copy Markdown
Contributor Author

@cPu1 cPu1 Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's required. ServicesV2 is an embedded field of ProviderServices.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just put everything directly on the struct? it will be easier.

Also all the other PRs will just replace the existing apis with the new ones, STS is one of the few where it will live alongside the V1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a new struct because the service methods (STS() etc.) are implemented differently in that the service client is configured in the method itself. Additionally, there's a lot more happening in pkg/eks/api.go, so I wanted this to be cleaner and keep the diff minimal. I wouldn't mind adding it to the existing struct though, but this PR has already grown too much.

Copy link
Copy Markdown
Contributor

@Himangini Himangini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@cPu1 cPu1 force-pushed the aws-sdk-v2 branch 2 times, most recently from 32506b1 to b532372 Compare March 22, 2022 11:05
Comment on lines +14 to +58
// ServicesV2 implements api.ServicesV2
type ServicesV2 struct {
config aws.Config

mu sync.Mutex
sts *sts.Client
cloudformation *cloudformation.Client
}

// STSV2 implements the AWS STS service
func (s *ServicesV2) STSV2() awsapi.STS {
s.mu.Lock()
defer s.mu.Unlock()
if s.sts == nil {
s.sts = sts.NewFromConfig(s.config, func(o *sts.Options) {
// Disable retryer for STS
// (see https://github.com/weaveworks/eksctl/issues/705)
o.Retryer = aws.NopRetryer{}
})
}
return s.sts
}

// CloudFormationV2 implements the AWS CloudFormation service
func (s *ServicesV2) CloudFormationV2() awsapi.CloudFormation {
s.mu.Lock()
defer s.mu.Unlock()
if s.cloudformation == nil {
s.cloudformation = cloudformation.NewFromConfig(s.config, func(o *cloudformation.Options) {
// Use adaptive mode for retrying CloudFormation requests to mimic
// the logic used for AWS SDK v1.
o.Retryer = retry.NewAdaptiveMode(func(o *retry.AdaptiveModeOptions) {
o.StandardOptions = []func(*retry.StandardOptions){
func(so *retry.StandardOptions) {
so.MaxAttempts = maxRetries
},
}
})
})
}
return s.cloudformation
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see both sides 😄

In reality, we don't need to worry too much about optimizing this area and creating all the services would be fine.

On the flip side, its a standard lazy initialization pattern. The one thing i would say is that we are assuming that all new services will use this pattern and that contributors will need to follow it. So, it might be worth documenting it somewhere?

@cPu1 cPu1 force-pushed the aws-sdk-v2 branch 5 times, most recently from 7396fb0 to f0e595f Compare March 23, 2022 09:12
@cPu1
Copy link
Copy Markdown
Contributor Author

cPu1 commented Mar 23, 2022

In reality, we don't need to worry too much about optimizing this area and creating all the services would be fine.

True, but this change was not just about optimizing code in terms of CPU cycles, but also to communicate the intent that only a few of these services will be used by most code paths.

On the flip side, its a standard lazy initialization pattern. The one thing i would say is that we are assuming that all new services will use this pattern and that contributors will need to follow it. So, it might be worth documenting it somewhere?

Agreed. I have added some comments to this effect.

Copy link
Copy Markdown
Contributor

@richardcase richardcase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still some difference of opinions on some things and few unresolved comments. From my side i see nothing that is blocking, especially as we have a number of follow-up issues and can revisit things if needed. So with that in mind this /lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tech-debt Leftover improvements in code, testing and building dependencies Pull requests that update a dependency file kind/improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate interfaces for AWS SDK v2 services from structs migrate to aws-sdk-go-v2 [1/9]

6 participants