Skip to content

Commit 0b6b47d

Browse files
cqgauravGaurav Vijaydisqhermanschaaferezrokah
authored
feat(crowdstrike): CrowdStrike source plugin (#5122)
<!-- 🎉 Thank you for making CloudQuery awesome by submitting a PR 🎉 --> #### Summary This PR adds a few services for a crowdstrike cloudquery plugin. It is intended to serve as a smaller example for adding a realisting CQ plugin. Tested using the following source and destination configs `cloudquery sync . ` // source.yml ``` kind: source spec: name: crowdstrike-0 # registry: github # path: cloudquery/aws registry: grpc path: localhost:7777 # version: v7.0.1 tables: ['*'] skip_tables: ['crowdstrike_incidents_crowdscore'] # skipped since this is pending on crowdstrike support. destinations: ["postgresql"] spec: client_id: "YOUR_CLIENT_ID" client_secret: "YOUR_CLIENT_SECRET" ``` ``` kind: destination spec: name: postgresql path: cloudquery/postgresql version: v1.7.9 write_mode: append spec: connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable" ``` Then start a local postgre in docker using `docker exec -it cloudquery_postgres psql -U postgres ` And test that the table was created. ``` postgres=# select * from crowdstrike_alerts_query; _cq_source_name | _cq_sync_time | _cq_id | _cq_parent _id | errors | meta | resources -----------------+----------------------------+--------------------------------------+----------- ----+--------+----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------- --------------------+----------- crowdstrike-0 | 2022-11-28 17:10:50.680223 | b0635155-2c56-41ee-8afb-59d7a0402ab2 | | null | {"writes": {"resources_affected": 0}, "trace_id": "173893ad-6264-4149-8a17-9c1e52a 8b5dc", "pagination": {"limit": 100, "total": 0, "offset": 0}, "powered_by": "detectsapi", "query _time": 0.01502205} | {} (1 row) ``` Note: I haven't been able to test this end to end because of the complexity of a test CrowdStrike setup combined with a non existent crowdstrike support, but hopefully it should work as an example. <!-- Use the following steps to ensure your PR is ready to be reviewed - [ ] Read the [contribution guidelines](../blob/main/CONTRIBUTING.md) 🧑‍🎓 - [ ] Test locally on your own infrastructure - [ ] Run `go fmt` to format your code 🖊 - [ ] Lint your changes via `golangci-lint run` 🚨 (install golangci-lint [here](https://golangci-lint.run/usage/install/#local-installation)) - [ ] Update or add tests 🧪 - [ ] Ensure the status checks below are successful ✅ ---> Co-authored-by: Gaurav Vijay <gauravvijay@fb.com> Co-authored-by: Kemal <223029+disq@users.noreply.github.com> Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com> Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>
1 parent 9385a69 commit 0b6b47d

39 files changed

Lines changed: 2223 additions & 2 deletions

‎.github/pr_labeler.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ azure:
44
- plugins/source/azure/**/*
55
cloudflare:
66
- plugins/source/cloudflare/**/*
7+
crowdstrike:
8+
- plugins/source/crowdstrike/**/*
79
digitalocean:
810
- plugins/source/digitalocean/**/*
911
datadog:

‎.github/styles/Vocab/Base/accept.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,11 +136,12 @@ Datadog
136136
nvm
137137
npm
138138
Gandi
139+
CrowdStrike
140+
crowdstrike
139141
parallelization
140142
hyperscale
141143
goroutines
142144
arn
143-
arns
144145
ARN
145146
ARNs
146147
SCP

‎.github/workflows/source_crowdstrike.yml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
name: Source Plugin CrowdStrike Workflow
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- "plugins/source/crowdstrike/**"
7+
- ".github/workflows/source_crowdstrike.yml"
8+
push:
9+
branches:
10+
- main
11+
paths:
12+
- "plugins/source/crowdstrike/**"
13+
- ".github/workflows/source_crowdstrike.yml"
14+
15+
jobs:
16+
resolve-runner:
17+
timeout-minutes: 5
18+
runs-on: ubuntu-latest
19+
outputs:
20+
runner: ${{ steps.resolve.outputs.runner }}
21+
steps:
22+
- name: Check if should use large runner
23+
id: large-runner
24+
# We want to speed runs on the main branch which prime the cache
25+
# We allow large runners only in this case to prevent forks from abusing them (it's enforced via runner groups access rules)
26+
# IF YOU WANT TO USE A LARGE RUNNER YOU NEED TO ADD THE WORKFLOW TO THE `CloudQuery releases` GROUP IN https://github.com/organizations/cloudquery/settings/actions/runner-groups
27+
if: github.event_name == 'push'
28+
run: |
29+
echo "runner=cloudquery-release-runner" >> $GITHUB_OUTPUT
30+
- name: Resolve runner
31+
id: resolve
32+
run: |
33+
RUNNER=${{ steps.large-runner.outputs.runner }}
34+
echo "runner=${RUNNER:-"ubuntu-latest"}" >> $GITHUB_OUTPUT
35+
plugins-source-crowdstrike:
36+
timeout-minutes: 30
37+
name: "plugins/source/crowdstrike"
38+
needs: [resolve-runner]
39+
runs-on: ${{ needs.resolve-runner.outputs.runner }}
40+
defaults:
41+
run:
42+
working-directory: ./plugins/source/crowdstrike
43+
steps:
44+
- uses: actions/checkout@v3
45+
with:
46+
fetch-depth: 2
47+
- name: Set up Go 1.x
48+
uses: actions/setup-go@v3
49+
with:
50+
go-version-file: plugins/source/crowdstrike/go.mod
51+
cache: true
52+
cache-dependency-path: plugins/source/crowdstrike/go.sum
53+
- name: golangci-lint
54+
uses: golangci/golangci-lint-action@v3
55+
with:
56+
version: v1.50.1
57+
working-directory: plugins/source/crowdstrike
58+
args: "--config ../../.golangci.yml"
59+
- name: Get dependencies
60+
run: go get -t -d ./...
61+
- name: Build
62+
run: go build .
63+
- name: Test
64+
run: make test
65+
- name: gen
66+
if: github.event_name == 'pull_request'
67+
run: make gen
68+
- name: Fail if generation updated files
69+
if: github.event_name == 'pull_request'
70+
run: test "$(git status -s | wc -l)" -eq 0
71+
validate-release:
72+
timeout-minutes: 30
73+
needs: [resolve-runner]
74+
runs-on: ${{ needs.resolve-runner.outputs.runner }}
75+
env:
76+
CGO_ENABLED: 0
77+
steps:
78+
- name: Checkout
79+
if: startsWith(github.head_ref, 'release-please--branches--main--components') || github.event_name == 'push'
80+
uses: actions/checkout@v3
81+
- uses: actions/cache@v3
82+
if: startsWith(github.head_ref, 'release-please--branches--main--components') || github.event_name == 'push'
83+
with:
84+
path: |
85+
~/.cache/go-build
86+
~/go/pkg/mod
87+
key: ${{ runner.os }}-go-1.19.3-release-cache-${{ hashFiles('plugins/source/crowdstrike/go.sum') }}
88+
restore-keys: |
89+
${{ runner.os }}-go-1.19.3-release-cache-plugins-source-crowdstrike
90+
- name: Set up Go
91+
if: startsWith(github.head_ref, 'release-please--branches--main--components') || github.event_name == 'push'
92+
uses: actions/setup-go@v3
93+
with:
94+
go-version-file: plugins/source/crowdstrike/go.mod
95+
- name: Install GoReleaser
96+
if: startsWith(github.head_ref, 'release-please--branches--main--components') || github.event_name == 'push'
97+
uses: goreleaser/goreleaser-action@v3
98+
with:
99+
distribution: goreleaser-pro
100+
version: latest
101+
install-only: true
102+
- name: Run GoReleaser Dry-Run
103+
if: startsWith(github.head_ref, 'release-please--branches--main--components') || github.event_name == 'push'
104+
run: goreleaser release --snapshot --rm-dist --skip-validate --skip-publish --skip-sign -f ./plugins/source/crowdstrike/.goreleaser.yaml
105+
env:
106+
GORELEASER_KEY: ${{ secrets.GORELEASER_KEY }}

‎.github/workflows/wait_for_required_workflows.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ jobs:
4343
"plugins/source/aws",
4444
"plugins/source/azure",
4545
"plugins/source/cloudflare",
46+
"plugins/source/crowdstrike",
4647
"plugins/source/digitalocean",
4748
"plugins/source/datadog",
4849
"plugins/source/gandi",

‎plugins/source/crowdstrike/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
crowdstrike

‎plugins/source/crowdstrike/.goreleaser.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
variables:
2+
component: source/crowdstrike
3+
binary: crowdstrike
4+
5+
project_name: plugins/source/crowdstrike
6+
7+
monorepo:
8+
tag_prefix: plugins-source-crowdstrike-
9+
dir: plugins/source/crowdstrike
10+
11+
includes:
12+
- from_file:
13+
# Relative to the directory Go Releaser is run from (which is the root of the repository)
14+
path: ./plugins/.goreleaser.yaml

‎plugins/source/crowdstrike/Makefile‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Generate mocks for mock/unit testing
2+
.PHONY: gen-mocks
3+
gen-mocks:
4+
go generate ./client/...
5+
6+
# Test unit
7+
.PHONY: test
8+
test:
9+
go test -timeout 3m ./...
10+
11+
# Install tools
12+
.PHONY: install-tools
13+
install-tools:
14+
@echo Installing tools from tools/tool.go
15+
@cat tools/tool.go | grep _ | awk -F'"' '{print $$2}' | xargs -tI % go install %
16+
17+
# Install pre-commit hooks. This requires pre-commit to be installed (https://pre-commit.com/)
18+
.PHONY: install-hooks
19+
install-hooks:
20+
pre-commit install
21+
22+
.PHONY: gen-docs
23+
gen-docs:
24+
rm -rf ./docs/tables/*
25+
go run main.go doc ./docs/tables
26+
sed 's_(\(.*\))_(https://github.com/cloudquery/cloudquery/blob/main/plugins/source/crowdstrike/docs/tables/\1)_' docs/tables/README.md > ../../../website/pages/docs/plugins/sources/crowdstrike/tables.md
27+
28+
.PHONY: lint
29+
lint:
30+
golangci-lint run --config ../../.golangci.yml
31+
32+
.PHONY: gen-code
33+
gen-code:
34+
grep -rl '// Code generated by codegen; DO NOT EDIT.' resources/services/* | xargs rm
35+
go run codegen/main.go
36+
37+
# All gen targets
38+
.PHONY: gen
39+
gen: gen-code gen-docs

‎plugins/source/crowdstrike/README.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# CrowdStrike Plugin
2+
3+
This plugin pulls information from CrowdStrike and loads it into any supported CloudQuery destination (e.g. PostgreSQL).
4+
5+
## Links
6+
7+
- [Tables](./docs/tables/README.md)
8+
9+
## Authentication
10+
11+
In order to fetch information from CrowdStrike, `cloudquery` needs to be authenticated. A client id and secret is required for authentication. Follow [these steps](https://www.crowdstrike.com/blog/tech-center/get-access-falcon-apis/) to set these up. Note that you will also need to enlist the client to have the appropriate scope for what you want to query.
12+
13+
## Configuration
14+
15+
To configure CloudQuery to extract from CrowdStrike, create a `.yml` file in your CloudQuery configuration directory.
16+
For example, the following configuration will extract information from CrowdStrike, and connect it to a `postgresql` destination plugin
17+
18+
```yaml
19+
kind: source
20+
spec:
21+
# Source spec section
22+
name: crowdstrike
23+
path: cloudquery/crowdstrike
24+
version: "0.0.1" # latest version of crowdstrike plugin
25+
tables: ["*"]
26+
destinations: ["postgresql"]
27+
spec:
28+
client_id: <CLIENT_ID>
29+
client_secret: <CLIENT_SECRET>
30+
```
31+
32+
## Example
33+
34+
You can reduce alert fatigue by narrowing alerts down from CrowdStrike using fuzzy matching.
35+
36+
```sql
37+
select * from crowdstrike_alerts_query where resources like ('%filter_here%');
38+
```

‎plugins/source/crowdstrike/client/client.go‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
package client
2+
3+
import (
4+
"context"
5+
"errors"
6+
"fmt"
7+
"os"
8+
9+
"github.com/cloudquery/plugin-sdk/schema"
10+
"github.com/cloudquery/plugin-sdk/specs"
11+
"github.com/crowdstrike/gofalcon/falcon"
12+
"github.com/rs/zerolog"
13+
"github.com/rs/zerolog/log"
14+
)
15+
16+
type Client struct {
17+
logger zerolog.Logger
18+
spec specs.Source
19+
Services Services
20+
}
21+
22+
type Services struct {
23+
Incidents Incidents
24+
Alerts Alerts
25+
}
26+
27+
func (*Client) Logger() *zerolog.Logger {
28+
return &log.Logger
29+
}
30+
31+
func (c *Client) ID() string {
32+
return c.spec.Name
33+
}
34+
35+
func New(ctx context.Context, logger zerolog.Logger, s specs.Source) (schema.ClientMeta, error) {
36+
crowdStrikeSpec := &Spec{}
37+
if err := s.UnmarshalSpec(&crowdStrikeSpec); err != nil {
38+
return nil, fmt.Errorf("failed to unmarshal CrowdStrike spec: %w", err)
39+
}
40+
clientId, ok := os.LookupEnv("FALCON_CLIENT_ID")
41+
if !ok {
42+
if crowdStrikeSpec.ClientID == "" {
43+
return nil, errors.New("missing FALCON_CLIENT_ID, either set it as an environment variable or pass it in the configuration")
44+
}
45+
clientId = crowdStrikeSpec.ClientID
46+
}
47+
48+
secret, ok := os.LookupEnv("FALCON_CLIENT_SECRET")
49+
if !ok {
50+
if crowdStrikeSpec.ClientID == "" {
51+
return nil, errors.New("missing FALCON_CLIENT_SECRET, either set it as an environment variable or pass it in the configuration")
52+
}
53+
secret = crowdStrikeSpec.ClientSecret
54+
}
55+
56+
c, err := falcon.NewClient(&falcon.ApiConfig{
57+
ClientId: clientId,
58+
ClientSecret: secret,
59+
Context: ctx,
60+
})
61+
if err != nil {
62+
return nil, err
63+
}
64+
return &Client{
65+
logger: logger,
66+
Services: Services{
67+
Incidents: c.Incidents,
68+
Alerts: c.Alerts,
69+
},
70+
spec: s,
71+
}, nil
72+
}

0 commit comments

Comments
 (0)