Skip to content

Commit 28d6dcd

Browse files
erezrokaherezrokah
authored
feat(gcp-resources): Add Cloud IoT (#6355)
 #### Summary Relevant API https://cloud.google.com/iot/docs/reference/cloudiot/rest ### Notes - We need to set a location to get registries, couldn't find an automatic way to list those (not even in the old API), so those are hard coded - `ListDevices` only provides `id` and `num_id` fields by default, so we need to set a field mask to get all the other fields. I couldn't find a way to "wildcard" all fields, so I'm using the table columns information to generate the mask - `DeviceConfig` and `DeviceState` don't have a built-in field that can be used as a primary key, so I added the parent `device_name` as one - This service is designated to be retired on August 16, 2023. I still think we should add it since the code works and it can help people migrate off the service <!--
1 parent afa651e commit 28d6dcd

19 files changed

+705
-0
lines changed

‎plugins/source/gcp/codegen/recipes/cloudiot.go‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
package recipes
2+
3+
import (
4+
iot "cloud.google.com/go/iot/apiv1"
5+
pb "cloud.google.com/go/iot/apiv1/iotpb"
6+
"github.com/cloudquery/plugin-sdk/codegen"
7+
"github.com/cloudquery/plugin-sdk/schema"
8+
)
9+
10+
func init() {
11+
resources := []*Resource{
12+
{
13+
SubService: "device_registries",
14+
Struct: &pb.DeviceRegistry{},
15+
PrimaryKeys: []string{ProjectIdColumn.Name, "name"},
16+
RequestStructFields: `Parent: "projects/" + c.ProjectId + "/locations/-",`,
17+
Relations: []string{"Devices()"},
18+
Description: "https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries#DeviceRegistry",
19+
},
20+
{
21+
SubService: "devices",
22+
Struct: &pb.Device{},
23+
PrimaryKeys: []string{ProjectIdColumn.Name, "name"},
24+
RequestStructFields: `Parent: parent.Item.(*pb.DeviceRegistry).Name,`,
25+
Description: "https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices#Device",
26+
ChildTable: true,
27+
Relations: []string{"DeviceConfigs(), DeviceStates()"},
28+
},
29+
{
30+
SubService: "device_configs",
31+
Struct: &pb.DeviceConfig{},
32+
PrimaryKeys: []string{ProjectIdColumn.Name, "device_name"},
33+
ExtraColumns: []codegen.ColumnDefinition{{Name: "device_name", Type: schema.TypeString, Resolver: `schema.ParentColumnResolver("name")`}},
34+
RequestStructFields: `Name: parent.Item.(*pb.Device).Name,`,
35+
Description: "https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices.configVersions#DeviceConfig",
36+
ChildTable: true,
37+
},
38+
{
39+
SubService: "device_states",
40+
Struct: &pb.DeviceState{},
41+
PrimaryKeys: []string{ProjectIdColumn.Name, "device_name"},
42+
ExtraColumns: []codegen.ColumnDefinition{{Name: "device_name", Type: schema.TypeString, Resolver: `schema.ParentColumnResolver("name")`}},
43+
RequestStructFields: `Name: parent.Item.(*pb.Device).Name,`,
44+
Description: "https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices.states#DeviceState",
45+
ChildTable: true,
46+
},
47+
}
48+
49+
for _, resource := range resources {
50+
resource.Service = "cloudiot"
51+
resource.ServiceAPIOverride = "iot"
52+
resource.Template = "newapi_list"
53+
resource.MockTemplate = "newapi_list_grpc_mock"
54+
resource.ProtobufImport = "cloud.google.com/go/iot/apiv1/iotpb"
55+
resource.MockImports = []string{"cloud.google.com/go/iot/apiv1"}
56+
resource.NewFunction = iot.NewDeviceManagerClient
57+
resource.RegisterServer = pb.RegisterDeviceManagerServer
58+
resource.SkipFetch = true
59+
resource.SkipMock = true
60+
}
61+
62+
Resources = append(Resources, resources...)
63+
}

‎plugins/source/gcp/docs/tables/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,10 @@
8080
- [gcp_clouddeploy_job_runs](gcp_clouddeploy_job_runs.md)
8181
- [gcp_clouderrorreporting_error_group_stats](gcp_clouderrorreporting_error_group_stats.md)
8282
- [gcp_clouderrorreporting_error_events](gcp_clouderrorreporting_error_events.md)
83+
- [gcp_cloudiot_device_registries](gcp_cloudiot_device_registries.md)
84+
- [gcp_cloudiot_devices](gcp_cloudiot_devices.md)
85+
- [gcp_cloudiot_device_configs](gcp_cloudiot_device_configs.md)
86+
- [gcp_cloudiot_device_states](gcp_cloudiot_device_states.md)
8387
- [gcp_compute_addresses](gcp_compute_addresses.md)
8488
- [gcp_compute_autoscalers](gcp_compute_autoscalers.md)
8589
- [gcp_compute_backend_services](gcp_compute_backend_services.md)

‎plugins/source/gcp/docs/tables/gcp_cloudiot_device_configs.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Table: gcp_cloudiot_device_configs
2+
3+
https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices.configVersions#DeviceConfig
4+
5+
The composite primary key for this table is (**project_id**, **device_name**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_cloudiot_devices](gcp_cloudiot_devices.md).
10+
11+
## Columns
12+
13+
| Name | Type |
14+
| ------------- | ------------- |
15+
|_cq_source_name|String|
16+
|_cq_sync_time|Timestamp|
17+
|_cq_id|UUID|
18+
|_cq_parent_id|UUID|
19+
|project_id (PK)|String|
20+
|device_name (PK)|String|
21+
|version|Int|
22+
|cloud_update_time|Timestamp|
23+
|device_ack_time|Timestamp|
24+
|binary_data|IntArray|

‎plugins/source/gcp/docs/tables/gcp_cloudiot_device_registries.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Table: gcp_cloudiot_device_registries
2+
3+
https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries#DeviceRegistry
4+
5+
The composite primary key for this table is (**project_id**, **name**).
6+
7+
## Relations
8+
9+
The following tables depend on gcp_cloudiot_device_registries:
10+
- [gcp_cloudiot_devices](gcp_cloudiot_devices.md)
11+
12+
## Columns
13+
14+
| Name | Type |
15+
| ------------- | ------------- |
16+
|_cq_source_name|String|
17+
|_cq_sync_time|Timestamp|
18+
|_cq_id|UUID|
19+
|_cq_parent_id|UUID|
20+
|project_id (PK)|String|
21+
|id|String|
22+
|name (PK)|String|
23+
|event_notification_configs|JSON|
24+
|state_notification_config|JSON|
25+
|mqtt_config|JSON|
26+
|http_config|JSON|
27+
|log_level|String|
28+
|credentials|JSON|

‎plugins/source/gcp/docs/tables/gcp_cloudiot_device_states.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Table: gcp_cloudiot_device_states
2+
3+
https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices.states#DeviceState
4+
5+
The composite primary key for this table is (**project_id**, **device_name**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_cloudiot_devices](gcp_cloudiot_devices.md).
10+
11+
## Columns
12+
13+
| Name | Type |
14+
| ------------- | ------------- |
15+
|_cq_source_name|String|
16+
|_cq_sync_time|Timestamp|
17+
|_cq_id|UUID|
18+
|_cq_parent_id|UUID|
19+
|project_id (PK)|String|
20+
|device_name (PK)|String|
21+
|update_time|Timestamp|
22+
|binary_data|IntArray|

‎plugins/source/gcp/docs/tables/gcp_cloudiot_devices.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Table: gcp_cloudiot_devices
2+
3+
https://cloud.google.com/iot/docs/reference/cloudiot/rest/v1/projects.locations.registries.devices#Device
4+
5+
The composite primary key for this table is (**project_id**, **name**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_cloudiot_device_registries](gcp_cloudiot_device_registries.md).
10+
11+
The following tables depend on gcp_cloudiot_devices:
12+
- [gcp_cloudiot_device_configs](gcp_cloudiot_device_configs.md)
13+
- [gcp_cloudiot_device_states](gcp_cloudiot_device_states.md)
14+
15+
## Columns
16+
17+
| Name | Type |
18+
| ------------- | ------------- |
19+
|_cq_source_name|String|
20+
|_cq_sync_time|Timestamp|
21+
|_cq_id|UUID|
22+
|_cq_parent_id|UUID|
23+
|project_id (PK)|String|
24+
|id|String|
25+
|name (PK)|String|
26+
|num_id|Int|
27+
|credentials|JSON|
28+
|last_heartbeat_time|Timestamp|
29+
|last_event_time|Timestamp|
30+
|last_state_time|Timestamp|
31+
|last_config_ack_time|Timestamp|
32+
|last_config_send_time|Timestamp|
33+
|blocked|Bool|
34+
|last_error_time|Timestamp|
35+
|last_error_status|JSON|
36+
|config|JSON|
37+
|state|JSON|
38+
|log_level|String|
39+
|metadata|JSON|
40+
|gateway_config|JSON|

‎plugins/source/gcp/go.mod‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ require (
2121
cloud.google.com/go/errorreporting v0.3.0
2222
cloud.google.com/go/functions v1.9.0
2323
cloud.google.com/go/iam v0.9.0
24+
cloud.google.com/go/iot v1.4.0
2425
cloud.google.com/go/kms v1.6.0
2526
cloud.google.com/go/logging v1.6.1
2627
cloud.google.com/go/longrunning v0.3.0

‎plugins/source/gcp/go.sum‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ cloud.google.com/go/functions v1.9.0 h1:35tgv1fQOtvKqH/uxJMzX3w6usneJ0zXpsFr9KAV
6161
cloud.google.com/go/functions v1.9.0/go.mod h1:Y+Dz8yGguzO3PpIjhLTbnqV1CWmgQ5UwtlpzoyquQ08=
6262
cloud.google.com/go/iam v0.9.0 h1:bK6Or6mxhuL8lnj1i9j0yMo2wE/IeTO2cWlfUrf/TZs=
6363
cloud.google.com/go/iam v0.9.0/go.mod h1:nXAECrMt2qHpF6RZUZseteD6QyanL68reN4OXPw0UWM=
64+
cloud.google.com/go/iot v1.4.0 h1:Y9+oZT9jD4GUZzORXTU45XsnQrhxmDT+TFbPil6pRVQ=
65+
cloud.google.com/go/iot v1.4.0/go.mod h1:dIDxPOn0UvNDUMD8Ger7FIaTuvMkj+aGk94RPP0iV+g=
6466
cloud.google.com/go/kms v1.6.0 h1:OWRZzrPmOZUzurjI2FBGtgY2mB1WaJkqhw6oIwSj0Yg=
6567
cloud.google.com/go/kms v1.6.0/go.mod h1:Jjy850yySiasBUDi6KFUwUv2n1+o7QZFyuUJg6OgjA0=
6668
cloud.google.com/go/logging v1.6.1 h1:ZBsZK+JG+oCDT+vaxwqF2egKNRjz8soXiS6Xv79benI=

‎plugins/source/gcp/resources/plugin/autogen_tables.go‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎plugins/source/gcp/resources/services/cloudiot/device_configs.go‎

Lines changed: 55 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)