Skip to content

Commit 9b08d0e

Browse files
shimonp21shimonp21
authored
feat: Add GCP Web Security Scanner (#6674)
<!-- 🎉 Thank you for making CloudQuery awesome by submitting a PR 🎉 --> #### Summary <!-- Explain what problem this PR addresses --> <!-- Use the following steps to ensure your PR is ready to be reviewed - [ ] Read the [contribution guidelines](../blob/main/CONTRIBUTING.md) 🧑‍🎓 - [ ] Test locally on your own infrastructure - [ ] Run `go fmt` to format your code 🖊 - [ ] Lint your changes via `golangci-lint run` 🚨 (install golangci-lint [here](https://golangci-lint.run/usage/install/#local-installation)) - [ ] Update or add tests 🧪 - [ ] Ensure the status checks below are successful ✅ --->
1 parent 78fe44d commit 9b08d0e

18 files changed

Lines changed: 555 additions & 0 deletions

‎plugins/source/gcp/docs/tables/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,4 +137,8 @@
137137
- [gcp_sql_users](gcp_sql_users.md)
138138
- [gcp_storage_buckets](gcp_storage_buckets.md)
139139
- [gcp_storage_bucket_policies](gcp_storage_bucket_policies.md)
140+
- [gcp_websecurityscanner_scan_configs](gcp_websecurityscanner_scan_configs.md)
141+
- [gcp_websecurityscanner_scan_config_scan_runs](gcp_websecurityscanner_scan_config_scan_runs.md)
142+
- [gcp_websecurityscanner_scan_config_scan_run_findings](gcp_websecurityscanner_scan_config_scan_run_findings.md)
143+
- [gcp_websecurityscanner_scan_config_scan_run_crawled_urls](gcp_websecurityscanner_scan_config_scan_run_crawled_urls.md)
140144
- [gcp_workflows_workflows](gcp_workflows_workflows.md)

‎plugins/source/gcp/docs/tables/gcp_websecurityscanner_scan_config_scan_run_crawled_urls.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Table: gcp_websecurityscanner_scan_config_scan_run_crawled_urls
2+
3+
https://cloud.google.com/security-command-center/docs/reference/web-security-scanner/rest/v1/projects.scanConfigs.scanRuns.crawledUrls/list#CrawledUrl
4+
5+
The composite primary key for this table is (**project_id**, **scan_run_name**, **http_method**, **url**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_websecurityscanner_scan_config_scan_runs](gcp_websecurityscanner_scan_config_scan_runs.md).
10+
11+
## Columns
12+
13+
| Name | Type |
14+
| ------------- | ------------- |
15+
|_cq_source_name|String|
16+
|_cq_sync_time|Timestamp|
17+
|_cq_id|UUID|
18+
|_cq_parent_id|UUID|
19+
|project_id (PK)|String|
20+
|scan_run_name (PK)|String|
21+
|http_method (PK)|String|
22+
|url (PK)|String|
23+
|body|String|

‎plugins/source/gcp/docs/tables/gcp_websecurityscanner_scan_config_scan_run_findings.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Table: gcp_websecurityscanner_scan_config_scan_run_findings
2+
3+
https://cloud.google.com/security-command-center/docs/reference/web-security-scanner/rest/v1/projects.scanConfigs.scanRuns.findings
4+
5+
The composite primary key for this table is (**project_id**, **name**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_websecurityscanner_scan_config_scan_runs](gcp_websecurityscanner_scan_config_scan_runs.md).
10+
11+
## Columns
12+
13+
| Name | Type |
14+
| ------------- | ------------- |
15+
|_cq_source_name|String|
16+
|_cq_sync_time|Timestamp|
17+
|_cq_id|UUID|
18+
|_cq_parent_id|UUID|
19+
|project_id (PK)|String|
20+
|name (PK)|String|
21+
|finding_type|String|
22+
|severity|String|
23+
|http_method|String|
24+
|fuzzed_url|String|
25+
|body|String|
26+
|description|String|
27+
|reproduction_url|String|
28+
|frame_url|String|
29+
|final_url|String|
30+
|tracking_id|String|
31+
|form|JSON|
32+
|outdated_library|JSON|
33+
|violating_resource|JSON|
34+
|vulnerable_headers|JSON|
35+
|vulnerable_parameters|JSON|
36+
|xss|JSON|
37+
|xxe|JSON|

‎plugins/source/gcp/docs/tables/gcp_websecurityscanner_scan_config_scan_runs.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Table: gcp_websecurityscanner_scan_config_scan_runs
2+
3+
https://cloud.google.com/security-command-center/docs/reference/web-security-scanner/rest/v1/projects.scanConfigs.scanRuns
4+
5+
The composite primary key for this table is (**project_id**, **name**).
6+
7+
## Relations
8+
9+
This table depends on [gcp_websecurityscanner_scan_configs](gcp_websecurityscanner_scan_configs.md).
10+
11+
The following tables depend on gcp_websecurityscanner_scan_config_scan_runs:
12+
- [gcp_websecurityscanner_scan_config_scan_run_findings](gcp_websecurityscanner_scan_config_scan_run_findings.md)
13+
- [gcp_websecurityscanner_scan_config_scan_run_crawled_urls](gcp_websecurityscanner_scan_config_scan_run_crawled_urls.md)
14+
15+
## Columns
16+
17+
| Name | Type |
18+
| ------------- | ------------- |
19+
|_cq_source_name|String|
20+
|_cq_sync_time|Timestamp|
21+
|_cq_id|UUID|
22+
|_cq_parent_id|UUID|
23+
|project_id (PK)|String|
24+
|name (PK)|String|
25+
|execution_state|String|
26+
|result_state|String|
27+
|start_time|Timestamp|
28+
|end_time|Timestamp|
29+
|urls_crawled_count|Int|
30+
|urls_tested_count|Int|
31+
|has_vulnerabilities|Bool|
32+
|progress_percent|Int|
33+
|error_trace|JSON|
34+
|warning_traces|JSON|

‎plugins/source/gcp/docs/tables/gcp_websecurityscanner_scan_configs.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Table: gcp_websecurityscanner_scan_configs
2+
3+
https://cloud.google.com/security-command-center/docs/reference/web-security-scanner/rest/v1/projects.scanConfigs#resource:-scanconfig
4+
5+
The composite primary key for this table is (**project_id**, **name**).
6+
7+
## Relations
8+
9+
The following tables depend on gcp_websecurityscanner_scan_configs:
10+
- [gcp_websecurityscanner_scan_config_scan_runs](gcp_websecurityscanner_scan_config_scan_runs.md)
11+
12+
## Columns
13+
14+
| Name | Type |
15+
| ------------- | ------------- |
16+
|_cq_source_name|String|
17+
|_cq_sync_time|Timestamp|
18+
|_cq_id|UUID|
19+
|_cq_parent_id|UUID|
20+
|project_id (PK)|String|
21+
|name (PK)|String|
22+
|display_name|String|
23+
|max_qps|Int|
24+
|starting_urls|StringArray|
25+
|authentication|JSON|
26+
|user_agent|String|
27+
|blacklist_patterns|StringArray|
28+
|schedule|JSON|
29+
|export_to_security_command_center|String|
30+
|risk_level|String|
31+
|managed_scan|Bool|
32+
|static_ip_scan|Bool|
33+
|ignore_http_status_errors|Bool|

‎plugins/source/gcp/go.mod‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ require (
3434
cloud.google.com/go/secretmanager v1.9.0
3535
cloud.google.com/go/serviceusage v1.4.0
3636
cloud.google.com/go/storage v1.28.0
37+
cloud.google.com/go/websecurityscanner v1.4.0
3738
cloud.google.com/go/workflows v1.9.0
3839
github.com/cloudquery/plugin-sdk v1.24.1
3940
github.com/golang/mock v1.6.0

‎plugins/source/gcp/go.sum‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX
9696
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
9797
cloud.google.com/go/storage v1.28.0 h1:DLrIZ6xkeZX6K70fU/boWx5INJumt6f+nwwWSHXzzGY=
9898
cloud.google.com/go/storage v1.28.0/go.mod h1:qlgZML35PXA3zoEnIkiPLY4/TOkUleufRlu6qmcf7sI=
99+
cloud.google.com/go/websecurityscanner v1.4.0 h1:y7yIFg/h/mO+5Y5aCOtVAnpGUOgqCH5rXQ2Oc8Oq2+g=
100+
cloud.google.com/go/websecurityscanner v1.4.0/go.mod h1:ebit/Fp0a+FWu5j4JOmJEV8S8CzdTkAS77oDsiSqYWQ=
99101
cloud.google.com/go/workflows v1.9.0 h1:7Chpin9p50NTU8Tb7qk+I11U/IwVXmDhEoSsdccvInE=
100102
cloud.google.com/go/workflows v1.9.0/go.mod h1:ZGkj1aFIOd9c8Gerkjjq7OW7I5+l6cSvT3ujaO/WwSA=
101103
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=

‎plugins/source/gcp/resources/plugin/tables.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ import (
3535
"github.com/cloudquery/plugins/source/gcp/resources/services/serviceusage"
3636
"github.com/cloudquery/plugins/source/gcp/resources/services/sql"
3737
"github.com/cloudquery/plugins/source/gcp/resources/services/storage"
38+
"github.com/cloudquery/plugins/source/gcp/resources/services/websecurityscanner"
3839
"github.com/cloudquery/plugins/source/gcp/resources/services/workflows"
3940
)
4041

@@ -129,6 +130,7 @@ func PluginAutoGeneratedTables() []*schema.Table {
129130
serviceusage.Services(),
130131
sql.Instances(),
131132
storage.Buckets(),
133+
websecurityscanner.ScanConfigs(),
132134
workflows.Workflows(),
133135
}
134136
}

‎plugins/source/gcp/resources/services/websecurityscanner/crawled_urls.go‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
package websecurityscanner
2+
3+
import (
4+
pb "cloud.google.com/go/websecurityscanner/apiv1/websecurityscannerpb"
5+
"github.com/cloudquery/plugin-sdk/schema"
6+
"github.com/cloudquery/plugin-sdk/transformers"
7+
"github.com/cloudquery/plugins/source/gcp/client"
8+
)
9+
10+
func CrawledUrls() *schema.Table {
11+
return &schema.Table{
12+
Name: "gcp_websecurityscanner_scan_config_scan_run_crawled_urls",
13+
Description: `https://cloud.google.com/security-command-center/docs/reference/web-security-scanner/rest/v1/projects.scanConfigs.scanRuns.crawledUrls/list#CrawledUrl`,
14+
Resolver: fetchCrawledUrls,
15+
Multiplex: client.ProjectMultiplexEnabledServices("websecurityscanner.googleapis.com"),
16+
Transform: transformers.TransformWithStruct(&pb.CrawledUrl{}, client.Options()...),
17+
Columns: []schema.Column{
18+
{
19+
Name: "project_id",
20+
Type: schema.TypeString,
21+
Resolver: client.ResolveProject,
22+
CreationOptions: schema.ColumnCreationOptions{
23+
PrimaryKey: true,
24+
},
25+
},
26+
{
27+
Name: "scan_run_name",
28+
Type: schema.TypeString,
29+
Resolver: schema.ParentColumnResolver("name"),
30+
CreationOptions: schema.ColumnCreationOptions{
31+
PrimaryKey: true,
32+
},
33+
},
34+
{
35+
Name: "http_method",
36+
Type: schema.TypeString,
37+
Resolver: schema.PathResolver("HttpMethod"),
38+
CreationOptions: schema.ColumnCreationOptions{
39+
PrimaryKey: true,
40+
},
41+
},
42+
{
43+
Name: "url",
44+
Type: schema.TypeString,
45+
Resolver: schema.PathResolver("Url"),
46+
CreationOptions: schema.ColumnCreationOptions{
47+
PrimaryKey: true,
48+
},
49+
},
50+
},
51+
}
52+
}

‎plugins/source/gcp/resources/services/websecurityscanner/crawled_urls_fetch.go‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package websecurityscanner
2+
3+
import (
4+
"context"
5+
6+
websecurityscanner "cloud.google.com/go/websecurityscanner/apiv1"
7+
pb "cloud.google.com/go/websecurityscanner/apiv1/websecurityscannerpb"
8+
"github.com/cloudquery/plugin-sdk/schema"
9+
"github.com/cloudquery/plugins/source/gcp/client"
10+
"google.golang.org/api/iterator"
11+
)
12+
13+
func fetchCrawledUrls(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- any) error {
14+
c := meta.(*client.Client)
15+
parentItem := parent.Item.(*pb.ScanRun)
16+
17+
gcpClient, err := websecurityscanner.NewClient(ctx, c.ClientOptions...)
18+
if err != nil {
19+
return err
20+
}
21+
22+
it := gcpClient.ListCrawledUrls(ctx, &pb.ListCrawledUrlsRequest{
23+
Parent: parentItem.Name,
24+
}, c.CallOptions...)
25+
for {
26+
resp, err := it.Next()
27+
if err == iterator.Done {
28+
break
29+
}
30+
if err != nil {
31+
return err
32+
}
33+
34+
res <- resp
35+
}
36+
return nil
37+
}

0 commit comments

Comments
 (0)