Set up AWS SDK v2 and migrate STS service#4926
Conversation
b6186d4 to
491270a
Compare
| func addDotImport(code []byte, packageName string) error { | ||
| d := decorator.NewDecorator(token.NewFileSet()) | ||
| f, err := d.Parse(code) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| for _, dl := range f.Decls { | ||
| gd, ok := dl.(*dst.GenDecl) | ||
| if !ok { | ||
| logger.Warning("expected type %T; got %T", &dst.GenDecl{}, dl) | ||
| continue | ||
| } | ||
|
|
||
| if gd.Tok == token.IMPORT { | ||
| gd.Specs = append(gd.Specs, &dst.ImportSpec{ | ||
| Name: &dst.Ident{ | ||
| Name: ".", | ||
| }, | ||
| Path: &dst.BasicLit{Kind: token.STRING, Value: strconv.Quote(packageName)}, | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| restorer := decorator.NewRestorer() | ||
| var buf bytes.Buffer | ||
| if err := restorer.Fprint(&buf, f); err != nil { | ||
| return fmt.Errorf("error restoring source: %w", err) | ||
| } | ||
| src, err := format.Source(buf.Bytes()) | ||
| if err != nil { | ||
| return fmt.Errorf("error formatting source: %w", src) | ||
| } | ||
| if _, err := os.Stdout.Write(src); err != nil { | ||
| return fmt.Errorf("error writing source to STDOUT: %w", err) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| func main() { | ||
| if len(os.Args) != 2 { | ||
| panic(fmt.Sprintf("usage: add_import <package-to-import>")) | ||
| } | ||
| source, err := ioutil.ReadAll(os.Stdin) | ||
| if err != nil { | ||
| panic(fmt.Errorf("unexpected error reading from STDIN: %w", err)) | ||
| } | ||
|
|
||
| packageName := os.Args[1] | ||
| if err = addDotImport(source, packageName); err != nil { | ||
| panic(err) | ||
| } | ||
| } |
There was a problem hiding this comment.
are these 73 lines of code really easier than just having the 1 sed 🤔 ?
There was a problem hiding this comment.
It's by no means easier than sed, but it's more robust IMO. We can also modify this program later to avoid using dot imports by qualifying local types with the package name.
There was a problem hiding this comment.
We can also modify this program later to avoid using dot imports by qualifying local types with the package name.
do we need to do that though 🤷 ?
It's by no means easier than sed, but it's more robust IMO.
I would disagree, the contents of addImport is pretty obscure, on a first pass I struggle to understand what its disagree. The sed is just a bit simpler to read, and much easier to test manually
There was a problem hiding this comment.
it's more robust IMO
I'm trying to understand the pros/cons of each method, sed vs addDotImport. @cPu1 could you expand a bit on this, please? addDotImport adds quite a lot of complexity compared to sed. What do you see as the issues or limitations of sed, and how does addDotImport solve this and perform better?
If we do go with this program, I would suggest adding at least the minimum amount of unit tests to ensure that it really is more robust, that it does what we expect it to do, and to make it more maintainable.
There was a problem hiding this comment.
@nikimanoledaki sed is not the right tool to manipulate Go sources. That sed snippet can generate programs that have syntax errors.
Here's what's wrong with using that sed snippet:
$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
// Code generated by ifacemaker; DO NOT EDIT.
package awsapi
import "context"
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
EOF
// Code generated by ifacemaker; DO NOT EDIT.
package awsapi
import "context"
. "github.com/aws/aws-sdk-go-v2/service/eks"
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
// Code generated by ifacemaker; DO NOT EDIT.
// Copyright 2020 Weaveworks.
// The awsapi package provides an interface to AWS services. The "context" package is used to carry deadlines and
// cancellation signals.
package awsapi
import (
"context"
)
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
EOF
// Code generated by ifacemaker; DO NOT EDIT.
// Copyright 2020 Weaveworks.
// The awsapi package provides an interface to AWS services. The "context" package is used to carry deadlines and
. "github.com/aws/aws-sdk-go-v2/service/eks"
// cancellation signals.
package awsapi
import (
"context"
. "github.com/aws/aws-sdk-go-v2/service/eks"
)
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
$ cat <<EOF | sed "/\"context\".*/a . \"github.com/aws/aws-sdk-go-v2/service/eks\""
package awsapi
import (
"fmt"
)
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
EOF
package awsapi
import (
"fmt"
)
// CloudFormation provides an interface to the AWS CloudFormation service.
type CloudFormation interface {
// ...
}
As for the maintenance of the add_import.go program, we have an issue for that: #4994
327b3c5 to
676ae71
Compare
1353f71 to
b5bccea
Compare
pkg/eks/services_v2.go
Outdated
| // ServicesV2 implements api.ServicesV2 | ||
| type ServicesV2 struct { | ||
| config aws.Config | ||
| } | ||
|
|
||
| // STSV2 implements the AWS STS service | ||
| func (s *ServicesV2) STSV2() awsapi.STS { | ||
| return sts.NewFromConfig(s.config, func(o *sts.Options) { | ||
| // Disable retryer for STS | ||
| // (see https://github.com/weaveworks/eksctl/issues/705) | ||
| o.Retryer = aws.NopRetryer{} | ||
| }) | ||
| } | ||
|
|
||
| // CloudFormationV2 implements the AWS CloudFormation service | ||
| func (s *ServicesV2) CloudFormationV2() awsapi.CloudFormation { | ||
| return cloudformation.NewFromConfig(s.config, func(o *cloudformation.Options) { | ||
| // Use adaptive mode for retrying CloudFormation requests to mimic | ||
| // the logic used for AWS SDK v1. | ||
| o.Retryer = retry.NewAdaptiveMode(func(o *retry.AdaptiveModeOptions) { | ||
| o.StandardOptions = []func(*retry.StandardOptions){ | ||
| func(so *retry.StandardOptions) { | ||
| so.MaxAttempts = maxRetries | ||
| }, | ||
| } | ||
| }) | ||
| }) | ||
| } |
There was a problem hiding this comment.
this can be removed right?
There was a problem hiding this comment.
It's required. ServicesV2 is an embedded field of ProviderServices.
There was a problem hiding this comment.
can we just put everything directly on the struct? it will be easier.
Also all the other PRs will just replace the existing apis with the new ones, STS is one of the few where it will live alongside the V1
There was a problem hiding this comment.
I created a new struct because the service methods (STS() etc.) are implemented differently in that the service client is configured in the method itself. Additionally, there's a lot more happening in pkg/eks/api.go, so I wanted this to be cleaner and keep the diff minimal. I wouldn't mind adding it to the existing struct though, but this PR has already grown too much.
32506b1 to
b532372
Compare
pkg/eks/services_v2.go
Outdated
| // ServicesV2 implements api.ServicesV2 | ||
| type ServicesV2 struct { | ||
| config aws.Config | ||
|
|
||
| mu sync.Mutex | ||
| sts *sts.Client | ||
| cloudformation *cloudformation.Client | ||
| } | ||
|
|
||
| // STSV2 implements the AWS STS service | ||
| func (s *ServicesV2) STSV2() awsapi.STS { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| if s.sts == nil { | ||
| s.sts = sts.NewFromConfig(s.config, func(o *sts.Options) { | ||
| // Disable retryer for STS | ||
| // (see https://github.com/weaveworks/eksctl/issues/705) | ||
| o.Retryer = aws.NopRetryer{} | ||
| }) | ||
| } | ||
| return s.sts | ||
| } | ||
|
|
||
| // CloudFormationV2 implements the AWS CloudFormation service | ||
| func (s *ServicesV2) CloudFormationV2() awsapi.CloudFormation { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| if s.cloudformation == nil { | ||
| s.cloudformation = cloudformation.NewFromConfig(s.config, func(o *cloudformation.Options) { | ||
| // Use adaptive mode for retrying CloudFormation requests to mimic | ||
| // the logic used for AWS SDK v1. | ||
| o.Retryer = retry.NewAdaptiveMode(func(o *retry.AdaptiveModeOptions) { | ||
| o.StandardOptions = []func(*retry.StandardOptions){ | ||
| func(so *retry.StandardOptions) { | ||
| so.MaxAttempts = maxRetries | ||
| }, | ||
| } | ||
| }) | ||
| }) | ||
| } | ||
| return s.cloudformation | ||
| } |
There was a problem hiding this comment.
I can see both sides 😄
In reality, we don't need to worry too much about optimizing this area and creating all the services would be fine.
On the flip side, its a standard lazy initialization pattern. The one thing i would say is that we are assuming that all new services will use this pattern and that contributors will need to follow it. So, it might be worth documenting it somewhere?
7396fb0 to
f0e595f
Compare
True, but this change was not just about optimizing code in terms of CPU cycles, but also to communicate the intent that only a few of these services will be used by most code paths.
Agreed. I have added some comments to this effect. |
richardcase
left a comment
There was a problem hiding this comment.
There are still some difference of opinions on some things and few unresolved comments. From my side i see nothing that is blocking, especially as we have a number of follow-up issues and can revisit things if needed. So with that in mind this /lgtm
Description
This changelist configures
aws-sdk-go-v2and adds interfaces and types for obtaining AWS services. A code generation script is added that, as part of the build process, generates interfaces from AWS SDK types for mocking. Due to this issue in ifacemaker, it passes the source generated by ifacemaker through another scriptadd_import.gothat adds the missing import.As part of #4877, the
STSservice is migrated to useaws-sdk-go-v2. One usage of STS cannot be migrated to v2 yet as it's blocked onsigs.k8s.io/aws-iam-authenticator/pkg/token.Closes #4925
Closes #4877
TODO:
Checklist
README.md, or theuserdocsdirectory)area/nodegroup) and kind (e.g.kind/improvement)BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯