Skip to content

[Security] Add security detection engine package#797

Merged
rw-access merged 14 commits intoelastic:masterfrom
rw-access:add-detection-rules-pkg
Mar 30, 2021
Merged

[Security] Add security detection engine package#797
rw-access merged 14 commits intoelastic:masterfrom
rw-access:add-detection-rules-pkg

Conversation

@rw-access
Copy link
Copy Markdown
Contributor

@rw-access rw-access commented Mar 22, 2021

What does this PR do?

Added an integration that contains security_rule assets to be used by the Detection Engine in Security. The core idea behind this effort is to use Fleet to update rules for the Detection Engine between releases. This way, users can get the latest released rules and package-relevant rules without needing to wait for the next stack release.

See the "related issues" section or ping me for more context. I'll be happy to point to additional (internal) design docs and discussions.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them. (have not tested the package using elastc-package yet)
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.

Author's Checklist

We've had a lot of conversations between teams, so I think we're in alignment. The name of the integration is TBD, and the specifics to install the security_rule asset type are also undecided.

How to test this PR locally

There is a chicken-and-egg issue here. Since it contains a brand new asset type, Kibana doesn't yet know how to install it. So the package will be published first as a pre-release and once this merges and is added to package-storage, then we can start developing the Kibana side to install the package.

There will be more iteration as we go between all of the repositories.

Related issues

*elastic/package-spec#142: Add security_rule asset type

Screenshots

None yet

@rw-access rw-access added the enhancement New feature or request label Mar 22, 2021
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Mar 22, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #797 updated

  • Start Time: 2021-03-30T20:30:21.395+0000

  • Duration: 45 min 10 sec

  • Commit: f9f5807

Test stats 🧪

Test Results
Failed 0
Passed 1842
Skipped 3
Total 1845

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 1

Expand to view the steps failures

Boot up the Elastic stack
  • Took 3 min 28 sec . View more details on here
  • Description: ../../build/elastic-package stack up -d -v

Log output

Expand to view the last 100 lines of log output

[2021-03-30T21:15:21.868Z] - kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.apiserver (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.container (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.controllermanager (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.event (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.node (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.pod (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.proxy (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.scheduler (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_container (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_cronjob (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_daemonset (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_deployment (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_node (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_persistentvolume (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_persistentvolumeclaim (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_pod (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_replicaset (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_resourcequota (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_service (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_statefulset (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_storageclass (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.system (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.volume (type: index_template)
[2021-03-30T21:15:21.869Z] Done
[2021-03-30T21:15:21.902Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations
[2021-03-30T21:15:22.199Z] + build/elastic-package stack dump -v --output build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Enable verbose logging
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Dump Elastic stack data
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Dump stack logs
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Recreate the output location (path: build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes)
[2021-03-30T21:15:22.200Z] 2021/03/30 21:15:21 DEBUG Dump stack logs for elasticsearch
[2021-03-30T21:15:22.200Z] 2021/03/30 21:15:21 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-03-30T21:15:23.155Z] 2021/03/30 21:15:23 DEBUG Dump stack logs for elastic-agent
[2021-03-30T21:15:23.155Z] 2021/03/30 21:15:23 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-03-30T21:15:24.548Z] 2021/03/30 21:15:24 DEBUG Dump stack logs for kibana
[2021-03-30T21:15:24.548Z] 2021/03/30 21:15:24 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-03-30T21:15:25.493Z] 2021/03/30 21:15:25 DEBUG Dump stack logs for package-registry
[2021-03-30T21:15:25.493Z] 2021/03/30 21:15:25 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-03-30T21:15:26.437Z] Path to stack dump: build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes
[2021-03-30T21:15:26.437Z] Done
[2021-03-30T21:15:26.459Z] Archiving artifacts
[2021-03-30T21:15:26.769Z] + build/elastic-package stack down -v
[2021-03-30T21:15:26.769Z] 2021/03/30 21:15:26 DEBUG Enable verbose logging
[2021-03-30T21:15:26.769Z] Take down the Elastic stack
[2021-03-30T21:15:26.769Z] 2021/03/30 21:15:26 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack down
[2021-03-30T21:15:27.340Z] The ELASTICSEARCH_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] The KIBANA_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] The ELASTIC_AGENT_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_elastic-agent_1    ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_kibana_1           ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_package-registry_1 ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_elasticsearch_1    ... 
[2021-03-30T21:15:29.704Z] 
Stopping elastic-package-stack_elastic-agent_1    ... done

Stopping elastic-package-stack_kibana_1           ... done

Stopping elastic-package-stack_package-registry_1 ... done

Stopping elastic-package-stack_elasticsearch_1    ... done
Removing elastic-package-stack_elastic-agent_is_ready_1    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elastic-agent_1             ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_kibana_is_ready_1           ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elasticsearch_is_ready_1    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_kibana_1                    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_package-registry_is_ready_1 ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_package-registry_1          ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elasticsearch_1             ... 
[2021-03-30T21:15:29.705Z] 
Removing elastic-package-stack_kibana_is_ready_1           ... done

Removing elastic-package-stack_kibana_1                    ... done

Removing elastic-package-stack_elastic-agent_is_ready_1    ... done

Removing elastic-package-stack_package-registry_is_ready_1 ... done

Removing elastic-package-stack_package-registry_1          ... done

Removing elastic-package-stack_elasticsearch_is_ready_1    ... done

Removing elastic-package-stack_elasticsearch_1             ... done

Removing elastic-package-stack_elastic-agent_1             ... done
Removing network elastic-package-stack_default
[2021-03-30T21:15:30.280Z] Done
[2021-03-30T21:15:30.424Z] Stage "Update Package Storage" skipped due to earlier failure(s)
[2021-03-30T21:15:30.679Z] Running on worker-395930 in /var/lib/jenkins/workspace/gest-manager_integrations_PR-797
[2021-03-30T21:15:30.722Z] [INFO] getVaultSecret: Getting secrets
[2021-03-30T21:15:30.829Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-30T21:15:32.866Z] + chmod 755 generate-build-data.sh
[2021-03-30T21:15:32.866Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12 FAILURE 2710068
[2021-03-30T21:15:32.867Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/steps/?limit=10000 -o steps-info.json
[2021-03-30T21:15:36.010Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/tests/?status=FAILED -o tests-errors.json
[2021-03-30T21:15:36.711Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/log/ -o pipeline-log.txt

@mtojek mtojek self-requested a review March 23, 2021 09:23
mtojek
mtojek previously requested changes Mar 23, 2021
Copy link
Copy Markdown
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't review JSON files, as I don't have domain knowledge here and can't verify them thoroughly.

Please merge the latest master to load the latest dependency on elastic-package (it should start accepting new Kibana types).

Before merging this PR we need to make sure the PR is green.

@mtojek mtojek requested a review from ycombinator March 23, 2021 09:33
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@rw-access rw-access requested a review from mtojek March 24, 2021 21:36
require (
github.com/blang/semver v3.5.1+incompatible
github.com/elastic/elastic-package v0.0.0-20210323050956-96ac7e929cae
github.com/elastic/elastic-package v0.0.0-20210324160257-57698a40f3e3
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you pull this go mod change into a speparate PR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it here: #804 (spotted also a different issue). Once it's merged, please rebase this PR against master.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged in master and confirmed the go changes are gone.
Thanks @mtojek

@mtojek
Copy link
Copy Markdown
Contributor

mtojek commented Mar 25, 2021

Hm... I spotted a weird issue:

[2021-03-24T21:22:19.014Z] Error: checking package failed: linting package failed: found 3 validation errors:
[2021-03-24T21:22:19.014Z]    1. item [test-audit.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/audit/_dev/test/pipeline]
[2021-03-24T21:22:19.014Z]    2. item [test-firewall.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/firewall/_dev/test/pipeline]
[2021-03-24T21:22:19.014Z]    3. item [test-vpcflow.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/vpcflow/_dev/test/pipeline]

Will investigate

EDIT:

It seems that there are few files that are not aligned with spec, I will fix them and also bump up the dependency on elastic-package.

@mtojek
Copy link
Copy Markdown
Contributor

mtojek commented Mar 25, 2021

@rw-access Please rebase this branch again against the master branch. I merged the update on elastic-package, so above errors should disappear.

@rw-access rw-access dismissed mtojek’s stale review March 25, 2021 17:14

Test are passing

@rw-access
Copy link
Copy Markdown
Contributor Author

rw-access commented Mar 25, 2021

@mtojek CI is green now. thanks for pulling the deps into a separate PR.
do i have your 👍 to merge?

Looking into a few more things first. Might need to change the underlying JSON format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants