feat(aws): Support identitystore and ssoadmin by jhudson10x · Pull Request #3005 · cloudquery/cloudquery

jhudson10x · 2022-10-18T19:52:40Z

This PR is ready for review for merge.There remains ssoadmin resources to be added. I can add them as part of this PR or I can open a new PR for those resources. Please advise.

Summary

Add support for the IAM Identity Center (Successor to AWS Single Sign-On) a.k.a ssoadmin and identitystore APIs

jhudson10x · 2022-10-18T19:53:40Z

refs #1418

plugins/source/aws/resources/services/ssoadmin/instances_fetch.go

erezrokah

Thanks @jhudson10x, added a few comments too?

What do you think about breaking this into multiple PRs per resource? Then it will be easier to merge the completed ones

plugins/source/aws/codegen/recipes/ssoadmin.go

plugins/source/aws/resources/services/identitystore/users_fetch.go

jhudson10x · 2022-10-24T18:38:12Z

Thanks @jhudson10x, added a few comments too?

What do you think about breaking this into multiple PRs per resource? Then it will be easier to merge the completed ones

I am ok with breaking these into multiple PRs. I apologize for the delay in wrapping this one up. Last week was very busy with an onsite meeting and did not get a chance to do much.

jhudson10x · 2022-10-25T18:03:00Z

Thanks @jhudson10x, added a few comments too?
What do you think about breaking this into multiple PRs per resource? Then it will be easier to merge the completed ones

I am ok with breaking these into multiple PRs. I apologize for the delay in wrapping this one up. Last week was very busy with an onsite meeting and did not get a chance to do much.

Just to be clear in case it wasn't from context - the identitystore service depends on the ssoadmin instance resource. This PR is ready for review for merge. I'm going to be finishing the remaining ssoadmin resources today, but if you want me to go ahead and wrap this one up I can open a new PR for those resources.

erezrokah

This is great work @jhudson10x, sorry for the late re-review.

This looks great pending a few comments. Please let me know what you think

plugins/source/aws/resources/plugin/tables.go

plugins/source/aws/resources/services/identitystore/identitystore_fetch.go

plugins/source/aws/resources/services/identitystore/users_fetch.go

erezrokah · 2022-10-27T12:02:00Z

plugins/source/aws/resources/services/ssoadmin/instances_fetch.go

+func fetchSsoadminInstances(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
+	svc := meta.(*client.Client).Services().SSOAdmin
+	config := ssoadmin.ListInstancesInput{}
+	response, err := svc.ListInstances(ctx, &config)


Does this API needs to be paginated like the others? It seems it supports that functionality

I think there is no possibility for more than 1 instance to exist. I will paginate it to be safe. It raises another question but we can look at that in next PR.

erezrokah · 2022-10-27T12:11:49Z

Additionally to get the CI passing you'll need to run make gen-docs in the AWS plugin directory and push any changes to the docs.

Finally if it makes sense, you can add mock tests to verify the fetch functions work as expected, see example(s) in https://github.com/cloudquery/cloudquery/blob/9a313bb24316b02538d2abde79bd0873bf81fe03/plugins/source/aws/resources/services/elbv2/load_balancers_mock_test.go

jhudson10x · 2022-10-31T14:54:38Z

@erezrokah I thought we could delay answering the question "is there ever more than 1 SSO instance in an account" but maybe we should address it now because I have assumed this in the change, e.g.:

+func getIamInstance(ctx context.Context, meta schema.ClientMeta) (types.InstanceMetadata, error) {                                                                                                                 
+       svc := meta.(*client.Client).Services().SSOAdmin                                                                                                                                                            
+       config := ssoadmin.ListInstancesInput{}                                                                                                                                                                     
+       response, err := svc.ListInstances(ctx, &config)                                                                                                                                                            
+       if err == nil {                                                                                                                                                                                             
+               for _, i := range response.Instances {                                                                                                                                                              
+                       return i, err                                                                                                                                                                               
+               }                                                                                                                                                                                                   
+       }                                                                                                                                                                                                           
+       return types.InstanceMetadata{}, err                                                                                                                                                                        
+}                                                                                                                                                                                                                  
+                                                                                                                                                                                                                   
+func fetchIdentitystoreGroups(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {                                                                                
+       instance, err := getIamInstance(ctx, meta)                                                                                                                                                                  
+       if err != nil {                                                                                                                                                                                             
+               return err                                                                                                                                                                                          
+       }     
... // use the returned instance to do fetch

FWIW the AWS docs clearly state you can only have one:

You can have only one identity source per organization in AWS Organizations.
https://docs.aws.amazon.com/singlesignon/latest/userguide/manage-your-identity-source.html

If there is a better way to access that identiy-store-id (this is what instance is needed for in above code) then I can change it. Maybe doing table relation? I do not know if that will work across services though.

#### Summary Related to #3326. This PRs changes the `/docs` command to a `/gen` command so it generates both code and docs. See example for the docs command in: #3005 (comment) c7f72fd We could have separate commands, but I think an update to code generation always requires a doc generation <!--

erezrokah

Thanks for the follow up digging into AWS docs 🙃

It all makes sense to me so I approved the PR. I'm planning to add mock tests for it today/tomorrow and then I'll merge it

SSOAdmin Instances table is passing validation but the resolver is not being called.

Probably needs its own dedicated region provider because looking at the AWS dashboard there appear to be more available regions than the 13 provided by the identitystore provider.

Need to determine appropriate method for using identity-store-id (from ssoadmin service instance resource) in the identitystore service resources.

Redundant information; all of it's contained in the group memberships table.

…ore_fetch.go

…h.go

🤖 I have created a release *beep* *boop* --- ## [4.6.0](plugins-source-aws-v4.5.0...plugins-source-aws-v4.6.0) (2022-11-06) ### Features * AppRunner add support for Connections ([#3602](#3602)) ([b6c17a2](b6c17a2)) * **aws:** AppRunner Resources: VPC Connector, VPC Ingress Connection And Autotscaling ([#3450](#3450)) ([f5cd42c](f5cd42c)) * **aws:** AppRunner support ObservabilityConfiguration ([#3603](#3603)) ([b93a66d](b93a66d)) * **aws:** Support identitystore and ssoadmin ([#3005](#3005)) ([afa463d](afa463d)) ### Bug Fixes * **aws:** Elasticache Engine Versions PK ([#3562](#3562)) ([59a7400](59a7400)), closes [#3561](#3561) * **deps:** Update plugin-sdk for aws to v0.13.20 ([#3569](#3569)) ([3876311](3876311)) * Fix bug in s3_cross_region_replication policy ([#3565](#3565)) ([515a7d0](515a7d0)) * Fix documentation ([#3608](#3608)) ([ea14f06](ea14f06)) * Update endpoints ([#3605](#3605)) ([20b9f4f](20b9f4f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

jhudson10x requested review from hermanschaaf and yevgenypats as code owners October 18, 2022 19:52

cq-bot added the aws label Oct 18, 2022

jhudson10x changed the title ~~WIP demonstrate SSOAdmin resource problem~~ WIP Support IdentityStore Oct 18, 2022

jhudson10x changed the title ~~WIP Support IdentityStore~~ WIP Support identitystore and ssoadmin Oct 18, 2022

jhudson10x changed the title ~~WIP Support identitystore and ssoadmin~~ feat(aws): support identitystore and ssoadmin Oct 18, 2022

jhudson10x changed the title ~~feat(aws): support identitystore and ssoadmin~~ WIP: feat(aws): support identitystore and ssoadmin Oct 18, 2022

vercel bot deployed to Preview – cloudquery-versions October 19, 2022 18:21 View deployment

vercel bot deployed to Preview – cloudquery-web October 19, 2022 18:21 View deployment

disq reviewed Oct 21, 2022

View reviewed changes

plugins/source/aws/resources/services/ssoadmin/instances_fetch.go Outdated Show resolved Hide resolved

disq reviewed Oct 21, 2022

View reviewed changes

plugins/source/aws/resources/services/ssoadmin/instances_fetch.go Outdated Show resolved Hide resolved

vercel bot deployed to Preview – cloudquery-versions October 23, 2022 09:10 View deployment

vercel bot deployed to Preview – cloudquery-web October 23, 2022 09:11 View deployment

erezrokah changed the title ~~WIP: feat(aws): support identitystore and ssoadmin~~ feat(aws): support identitystore and ssoadmin Oct 23, 2022

erezrokah requested changes Oct 23, 2022

View reviewed changes

plugins/source/aws/codegen/recipes/ssoadmin.go Show resolved Hide resolved

plugins/source/aws/resources/services/identitystore/users_fetch.go Outdated Show resolved Hide resolved

vercel bot deployed to Preview – cloudquery-versions October 24, 2022 19:49 View deployment

vercel bot deployed to Preview – cloudquery-web October 24, 2022 19:49 View deployment

jhudson10x mentioned this pull request Oct 25, 2022

Support IdentityStore #1418

Closed

vercel bot deployed to Preview – cloudquery-versions October 25, 2022 14:22 View deployment

vercel bot deployed to Preview – cloudquery-web October 25, 2022 14:25 View deployment

jhudson10x requested review from erezrokah and removed request for hermanschaaf and yevgenypats October 26, 2022 18:46

erezrokah changed the title ~~feat(aws): support identitystore and ssoadmin~~ feat(aws): Support identitystore and ssoadmin Oct 27, 2022

erezrokah requested changes Oct 27, 2022

View reviewed changes

vercel bot deployed to Preview – cloudquery-versions October 27, 2022 17:01 View deployment

erezrokah mentioned this pull request Oct 31, 2022

chore(ci): Change docs command to gen command #3345

Merged

yevgenypats added the resource label Nov 1, 2022

erezrokah force-pushed the 1418_support_identitystore branch from 95d65fa to 3afec60 Compare November 2, 2022 09:49

erezrokah approved these changes Nov 2, 2022

View reviewed changes

vercel bot deployed to Preview – cloudquery-versions November 2, 2022 17:38 View deployment

vercel bot deployed to Preview – cloudquery-web November 2, 2022 17:39 View deployment

jhudson10x and others added 11 commits November 3, 2022 19:01

WIP demonstrate SSOAdmin resource problem

e8a5832

SSOAdmin Instances table is passing validation but the resolver is not being called.

Update multiplexer to use an extant service

14dcc85

Switch to identitystore region provider.

3081d12

Probably needs its own dedicated region provider because looking at the AWS dashboard there appear to be more available regions than the 13 provided by the identitystore provider.

Add skeletons for sso groups/users

1e8f2a2

Need to determine appropriate method for using identity-store-id (from ssoadmin service instance resource) in the identitystore service resources.

Getting ready for cleanup

9142800

Remove user memberships

6ca6afb

Redundant information; all of it's contained in the group memberships table.

Update plugins/source/aws/resources/services/identitystore/identityst…

1bf2813

…ore_fetch.go

Update plugins/source/aws/resources/services/identitystore/users_fetc…

108ac32

…h.go

chore: Update docs

3800f4d

Incorporate PR feedback

56533a7

feat: Update with new code gen

23c5190

erezrokah force-pushed the 1418_support_identitystore branch from 1acf344 to 23c5190 Compare November 3, 2022 17:08

Add Mock tests

f1a2dd5

bbernays self-requested a review November 3, 2022 19:23

bbernays approved these changes Nov 3, 2022

View reviewed changes

Merge branch 'main' into 1418_support_identitystore

9c91977

bbernays added the automerge Automatically merge once required checks pass label Nov 3, 2022

vercel bot deployed to Preview – cloudquery-web November 3, 2022 19:25 View deployment

vercel bot deployed to Preview – cloudquery-versions November 3, 2022 19:25 View deployment

kodiakhq bot merged commit afa463d into cloudquery:main Nov 3, 2022

cq-bot mentioned this pull request Nov 3, 2022

chore(main): Release plugins-source-aws v4.6.0 #3560

Merged

Conversation

jhudson10x commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

jhudson10x commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

erezrokah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jhudson10x commented Oct 24, 2022

Uh oh!

jhudson10x commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erezrokah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erezrokah Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

jhudson10x Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erezrokah commented Oct 27, 2022

Uh oh!

jhudson10x commented Oct 31, 2022

Uh oh!

erezrokah left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jhudson10x commented Oct 18, 2022 •

edited

Loading

jhudson10x commented Oct 25, 2022 •

edited

Loading

jhudson10x Oct 27, 2022 •

edited

Loading