feat(gcp-resources): Add Vertex-AI by erezrokah · Pull Request #5762 · cloudquery/cloudquery

erezrokah · 2022-12-18T20:37:43Z

Summary

PR to add Vertex-AI resources.

Those require:

Explicitly setting the API endpoint to one of those listed in https://cloud.google.com/vertex-ai/docs/reference/rest#service-endpoint (the API throws errors without it, see aiplatform: Prediction error googleapis/google-cloud-go#6649 (comment)) to list the locations for each resource.
Listing the locations before calling the actual list function (with the correct endpoint for that location), hence the relations.

Caveats

Some API calls fail out of the box, but that happens to gcloud too 😕 I filtered these locations manually for now

github-actions · 2022-12-18T20:38:14Z

This PR has the following changes to source plugin(s) tables:

Table gcp_aiplatform_batch_prediction_jobs was added
Table gcp_aiplatform_custom_jobs was added
Table gcp_aiplatform_data_labeling_jobs was added
Table gcp_aiplatform_dataset_locations was added
Table gcp_aiplatform_datasets was added
Table gcp_aiplatform_endpoint_locations was added
Table gcp_aiplatform_endpoints was added
Table gcp_aiplatform_featurestore_locations was added
Table gcp_aiplatform_featurestores was added
Table gcp_aiplatform_hyperparameter_tuning_jobs was added
Table gcp_aiplatform_index_endpoints was added
Table gcp_aiplatform_index_locations was added
Table gcp_aiplatform_indexendpoint_locations was added
Table gcp_aiplatform_indexes was added
Table gcp_aiplatform_job_locations was added
Table gcp_aiplatform_metadata_locations was added
Table gcp_aiplatform_metadata_stores was added
Table gcp_aiplatform_model_deployment_monitoring_jobs was added
Table gcp_aiplatform_model_locations was added
Table gcp_aiplatform_models was added
Table gcp_aiplatform_operations was added
Table gcp_aiplatform_pipeline_jobs was added
Table gcp_aiplatform_pipeline_locations was added
Table gcp_aiplatform_specialist_pools was added
Table gcp_aiplatform_specialistpool_locations was added
Table gcp_aiplatform_studies was added
Table gcp_aiplatform_tensorboard_locations was added
Table gcp_aiplatform_tensorboards was added
Table gcp_aiplatform_training_pipelines was added
Table gcp_aiplatform_vizier_locations was added

erezrokah · 2022-12-19T14:22:05Z

~~Blocked by #5803~~

erezrokah · 2022-12-19T14:22:48Z

plugins/source/gcp/codegen/recipes/aiplatform.go

+)
+
+func init() {
+	resources := []*Resource{


This is the bulk of the PR, the other are changes to support listing locations before listing the actual resources.

These are the locations.*.list top level resources from https://cloud.google.com/vertex-ai/docs/reference/rest.

The resources to list the locations are generated based on these.
To get the locations we need to use the specific client for that resource, for example (&aiplatform.JobClient{}).ListLocations != (&aiplatform.DatasetClient{}).ListLocations

erezrokah · 2022-12-19T14:28:35Z

plugins/source/gcp/codegen/templates/newapi_list.go.tpl

-	gcpClient, err := {{.Service}}.{{.NewFunctionName}}(ctx, c.ClientOptions...)
+	{{ if .ClientOptions }}
+	clientOptions := c.ClientOptions
+	clientOptions = append([]option.ClientOption{ {{ range .ClientOptions }}{{.}},{{end}} }, clientOptions...)


We prepend the options, otherwise the tests won't be able to override the endpoint URL

erezrokah · 2022-12-20T17:11:06Z

plugins/source/gcp/codegen/recipes/aiplatform.go

+}
+
+func isFieldMockable(field reflect.StructField) bool {
+	nonMockables := []any{&structpb.Value{}, &structpb.Struct{}, &pb.Model{}, &pb.PipelineJob_RuntimeConfig{}}


Faker can't mock these as we need to set the fields in a specific way so they can be MarshalJSONed

erezrokah · 2022-12-20T17:15:28Z

plugins/source/gcp/codegen/main.go

+	return responseHasNextPage
+}
+
+func generateMockTestData(r *recipes.Resource) {


This function creates the data necessary to mock the relations gRPC server calls.
We create a server implementation with all the List* methods, then register it.
See example in https://github.com/erezrokah/cloudquery/blob/863ced86bda2266a3d076b98ecd2f70ac1d23403/plugins/source/gcp/resources/services/aiplatform/job_locations_mock_test.go#L44

I think we can use this approach to get all list functions of a client to remove some boilerplate. I'll look into it separately

erezrokah · 2022-12-20T17:22:53Z

plugins/source/gcp/codegen/recipes/aiplatform.go

+
+		locationSubService := getLocationsServiceName(resource.NewFunction) + "_locations"
+		locationResource := locationsMap[locationSubService]
+		if locationResource == nil {


We need a location resource for all the resources under it, for example JobClient.ListLocations will allow us to list all resources under JobClient.
That's the reason need to check for an existing one, and append the resource as a relation (e.g. JobClient.ListLocations resource we have all relevant List* resources as relations)

erezrokah · 2022-12-20T17:23:33Z

plugins/source/gcp/codegen/recipes/base.go

 	// NameTransformer custom name transformer for resource
 	NameTransformer func(field reflect.StructField) (string, error)
+	// ClientOptions is a list of options to pass to the client
+	ClientOptions []string


We need ClientOptions so we can override the endpoint

erezrokah · 2022-12-20T17:24:38Z

plugins/source/gcp/codegen/recipes/base.go

+	ClientOptions []string
+	// IgnoreInTestsColumns is a list of columns to ignore in tests
+	IgnoreInTestsColumns []string
+	ParentFilterFunc     string


Apparently ListLocations returns invalid locations us and eu that can't be used to list sub resources, so we need to filter them. The function itself is manually written

vercel · 2022-12-21T08:39:39Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cloudquery-web	🔄 Building (Inspect)		Dec 26, 2022 at 1:56PM (UTC)

🤖 I have created a release *beep* *boop* --- ## [5.4.0](plugins-source-gcp-v5.3.1...plugins-source-gcp-v5.4.0) (2023-01-03) ### Features * **gcp-codegen:** Make ListFunction optional ([#5971](#5971)) ([5717f05](5717f05)) * **gcp-resources:** Add AppEngine resources ([#5972](#5972)) ([d6b6c6a](d6b6c6a)) * **gcp-resources:** Add Artifact Registry ([#6020](#6020)) ([76faa06](76faa06)) * **gcp-resources:** Add Bare Metal Solution ([#6044](#6044)) ([9a80b27](9a80b27)) * **gcp-resources:** Add Batch ([#6128](#6128)) ([43cd3be](43cd3be)) * **gcp-resources:** Add BeyondCorp ([#6133](#6133)) ([48aafd0](48aafd0)) * **gcp-resources:** Add Vertex-AI ([#5762](#5762)) ([d22b69d](d22b69d)) * **gcp:** Auto generate services list ([#5961](#5961)) ([0467de5](0467de5)) ### Bug Fixes * **deps:** Update google.golang.org/genproto digest to f9683d7 ([#6170](#6170)) ([0df29be](0df29be)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.14.0 ([#6025](#6025)) ([35b2cfc](35b2cfc)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.15.0 ([#6071](#6071)) ([684b525](684b525)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.15.1 ([#6079](#6079)) ([650659c](650659c)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.16.0 ([#6098](#6098)) ([7bacdf3](7bacdf3)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.16.1 ([#6214](#6214)) ([53b2415](53b2415)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.17.0 ([#6256](#6256)) ([b19f6cd](b19f6cd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

cq-bot added gcp labels Dec 18, 2022

vercel bot deployed to Preview December 18, 2022 20:39 View deployment

erezrokah force-pushed the feat/gcp_ai branch from dfdc686 to 2edc650 Compare December 18, 2022 20:44

vercel bot deployed to Preview December 18, 2022 20:45 View deployment

erezrokah force-pushed the feat/gcp_ai branch from 2edc650 to 369a9f3 Compare December 18, 2022 20:47

vercel bot deployed to Preview December 18, 2022 20:47 View deployment

erezrokah force-pushed the feat/gcp_ai branch from 369a9f3 to 35db1f7 Compare December 19, 2022 13:21

vercel bot deployed to Preview December 19, 2022 13:22 View deployment

erezrokah commented Dec 19, 2022

View reviewed changes

erezrokah changed the title ~~feat(gcp): Add Vertex-AI resources~~ feat(gcp): Add Vertex-AI top level resources Dec 19, 2022

erezrokah changed the title ~~feat(gcp): Add Vertex-AI top level resources~~ feat(gcp): Add Vertex-AI top level resources projects.locations.*.list Dec 19, 2022

erezrokah changed the title ~~feat(gcp): Add Vertex-AI top level resources projects.locations.*.list~~ feat(gcp): Add Vertex-AI top level resources - projects.locations.*.list Dec 19, 2022

erezrokah force-pushed the feat/gcp_ai branch from 35db1f7 to cdd90a0 Compare December 19, 2022 17:42

vercel bot deployed to Preview December 19, 2022 17:44 View deployment

erezrokah force-pushed the feat/gcp_ai branch 2 times, most recently from 4ca074f to 9b8c448 Compare December 20, 2022 15:21

vercel bot deployed to Preview December 20, 2022 16:13 View deployment

erezrokah force-pushed the feat/gcp_ai branch 2 times, most recently from 66d5d42 to 863ced8 Compare December 20, 2022 17:07

erezrokah commented Dec 20, 2022

View reviewed changes

erezrokah marked this pull request as ready for review December 20, 2022 18:09

erezrokah requested a review from yevgenypats as a code owner December 20, 2022 18:09

erezrokah requested a review from disq as a code owner December 20, 2022 18:09

erezrokah force-pushed the feat/gcp_ai branch from d4a1b66 to 12ac6f8 Compare December 21, 2022 10:07

vercel bot deployed to Preview December 21, 2022 10:08 View deployment

erezrokah force-pushed the feat/gcp_ai branch from 12ac6f8 to 1c95cac Compare December 22, 2022 12:47

erezrokah mentioned this pull request Dec 22, 2022

feat(gcp): Add missing GCP resources #5917

Closed

erezrokah force-pushed the feat/gcp_ai branch from 1c95cac to d8c5d0a Compare December 26, 2022 13:56

vercel bot deployed to Preview December 27, 2022 08:17 View deployment

erezrokah changed the title ~~feat(gcp): Add Vertex-AI top level resources - projects.locations.*.list~~ feat(gcp-resources): Add Vertex-AI Dec 27, 2022

erezrokah force-pushed the feat/gcp_ai branch 2 times, most recently from db15807 to fe62993 Compare December 28, 2022 16:50

yevgenypats approved these changes Dec 29, 2022

View reviewed changes

erezrokah added automerge Automatically merge once required checks pass and removed automerge Automatically merge once required checks pass labels Dec 29, 2022

vercel bot deployed to Preview December 29, 2022 09:37 View deployment

erezrokah added 4 commits December 29, 2022 12:00

feat(gcp): Add Vertex-AI resources

87d169a

fix: Update description for location tables

0bb35a4

refactor: Remove ServiceDNS

44600fd

chore: Clean up locations list method

d4f62f4

erezrokah force-pushed the feat/gcp_ai branch from 4fc1b6d to d4f62f4 Compare December 29, 2022 10:05

vercel bot deployed to Preview December 29, 2022 10:07 View deployment

erezrokah merged commit d22b69d into cloudquery:main Dec 29, 2022

erezrokah deleted the feat/gcp_ai branch December 29, 2022 10:16

cq-bot mentioned this pull request Dec 29, 2022

chore(main): Release plugins-source-gcp v5.4.0 #6057

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gcp-resources): Add Vertex-AI#5762

feat(gcp-resources): Add Vertex-AI#5762
erezrokah merged 4 commits intocloudquery:mainfrom
erezrokah:feat/gcp_ai

erezrokah commented Dec 18, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Dec 18, 2022 •

edited

Loading

Uh oh!

erezrokah commented Dec 19, 2022 •

edited

Loading

Uh oh!

erezrokah Dec 19, 2022 •

edited

Loading

Uh oh!

erezrokah Dec 19, 2022

Uh oh!

erezrokah Dec 20, 2022

Uh oh!

erezrokah Dec 20, 2022 •

edited

Loading

Uh oh!

erezrokah Dec 21, 2022

Uh oh!

erezrokah Dec 20, 2022 •

edited

Loading

Uh oh!

erezrokah Dec 20, 2022

Uh oh!

erezrokah Dec 20, 2022

Uh oh!

vercel bot commented Dec 21, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

erezrokah commented Dec 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Caveats

Uh oh!

github-actions bot commented Dec 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR has the following changes to source plugin(s) tables:

Uh oh!

erezrokah commented Dec 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erezrokah Dec 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 19, 2022

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

erezrokah Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

vercel bot commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

erezrokah commented Dec 18, 2022 •

edited

Loading

github-actions bot commented Dec 18, 2022 •

edited

Loading

erezrokah commented Dec 19, 2022 •

edited

Loading

erezrokah Dec 19, 2022 •

edited

Loading

erezrokah Dec 20, 2022 •

edited

Loading

erezrokah Dec 20, 2022 •

edited

Loading

vercel bot commented Dec 21, 2022 •

edited

Loading