[Security Solution] Detection rules bootstrap endpoint by xcrzx · Pull Request #189518 · elastic/kibana

xcrzx · 2024-07-30T14:05:32Z

Resolves: #187647

Summary

Added a new API endpoint POST /internal/detection_engine/prebuilt_rules/_bootstrap. This endpoint is responsible for installing the necessary packages for prebuilt detection rules to function properly. This allows us to avoid calling Fleet directly from FE and helps encapsulate complex logic of the package version selection in a single place on the backend.

Currently, it installs or upgrades (if already installed) two packages: endpoint and security_detection_engine.

The response looks like this:

{
  packages: [
    {
      name: 'detection_engine',
      version: '1.0.0',
      status: 'installed',
    },
    {
      name: 'endpoint',
      version: '1.0.0',
      status: 'already_installed',
    },
  ],
}

We call this endpoint from Kibana every time a user lands on any security solution page. The endpoint checks if the required packages are missing or if a newer version is available. If so, the newer version is installed, and the package status will be installed in the response. If all packages are up-to-date, the package status will be already_installed in the response. This allows us to invalidate prebuilt rule endpoints more efficiently and avoid sending extra requests from Kibana:

if (
  response?.packages.find((pkg) => pkg.name === PREBUILT_RULES_PACKAGE_NAME)?.status === 'installed'
) {
  // Invalidate other pre-packaged rules related queries. We need to do
  // that only if the prebuilt rules package was installed, indicating
  // that there might be new rules to install.
  invalidatePrePackagedRulesStatus();
  invalidatePrebuiltRulesInstallReview();
  invalidatePrebuiltRulesUpdateReview();
}

The performance gain is that we do not invalidate prebuilt rules when the package is already installed.

Previously:
Fetch rules initially -> Upgrade rules package -(always)-> Re-fetch rules

Now:
Fetch rules initially -> Upgrade rules package -(only if there's a new package version)-> Re-fetch rules

This will result in fewer redundant API requests from Kibana.

xcrzx · 2024-08-02T07:38:32Z

buildkite test this

xcrzx · 2024-08-02T08:41:46Z

buildkite test this

elasticmachine · 2024-08-02T10:54:21Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-08-02T10:54:23Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-08-02T10:54:24Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

elasticmachine · 2024-08-02T10:55:01Z

Pinging @elastic/fleet (Team:Fleet)

elasticmachine · 2024-08-02T10:55:01Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

juliaElastic

Fleet change LGTM

jpdjere · 2024-08-02T15:51:52Z

...ne/prebuilt_rules/api/install_prebuilt_rules_and_timelines/install_prebuilt_rules_package.ts

Why is this additional check needed? Would it otherwise be possible to get a prerelease version in serverless, somehow?

Prerelease packages are considered unstable, so should not be installed in Serverless. There's some context in this PR: #170975 (comment)

It should still be possible to install a prerelease version of a package by calling the Fleet API directly, if needed.

jpdjere · 2024-08-02T16:03:42Z

...ne/prebuilt_rules/api/install_prebuilt_rules_and_timelines/install_prebuilt_rules_package.ts

Is it not necessary here to check that Fleet is initialized?

This is a good question. On the front end, we were waiting for Fleet initialization before installing packages. However, on the backend, similar code simply doesn't exist, and I'm not sure why. I cannot find any initialization attempts or awaiting of initialization completion in the Fleet API. However, they do have a setup endpoint that is called from Kibana. I'll try to figure out if we need to call it before installing our packages.

tonyghiani

Obs change LGTM

maryam-saeidi · 2024-08-05T07:47:33Z

...bservability_solution/synthetics/server/routes/synthetics_service/install_index_templates.ts

What is this change about?
I am trying to understand if returning installation.package changes the rest of the flow since I only see this change related to our team 🤔

Previously, the package information was returned directly. Now, it is under the package prop. Therefore, no changes in the logic at this line.

I see, I think the confusing part for me was the installation.package type which is Installation.

I checked the installation type, and I saw that it is now EnsurePackageResult. We could rename the installation to avoid confusion (to something that matches the type name).

jpdjere · 2024-08-05T08:13:56Z

...ne/prebuilt_rules/api/install_prebuilt_rules_and_timelines/install_prebuilt_rules_package.ts

Under the hood, this endpoint still uses this Fleet method right? Just trying to understand if there's any performance gain here: you explained that some stuff that Fleet does on their side -like checking references- is not necessary for installing our pacakge. But I guess that's an improvement not for this PR, but when we get content-only packages?

Nothing changes regarding the package installation method, that's right. We will implement optimizations in package installation separately. The only performance gain in this PR is that we do not invalidate prebuilt rules when the package is already installed.

Previously:
Fetch rules initially -> Upgrade rules package -(always)-> Re-fetch rules

Now:
Fetch rules initially -> Upgrade rules package -(only if there's a new package version)-> Re-fetch rules

This will result in fewer redundant API requests from Kibana.

maryam-saeidi

LGTM

jpdjere

Thanks for the explanations, and checking about Fleet initialisation 👍 ✅

kibana-ci · 2024-08-07T09:46:38Z

💚 Build Succeeded

Buildkite Build
Commit: e66c663

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	5610	5609	-1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`securitySolution`	121	123	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	20.5MB	20.5MB	-2.8KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	73	74	+1

Unknown metric groups

API count

id	before	after	diff
`securitySolution`	190	192	+2

History

💔 Build #226039 failed 4db454cc04604ebb3e590d42b097177f297a0b96
💔 Build #225994 failed e6025bf6bf2bf2c5ac896ee339da7435702f7d2a
💛 Build #225554 was flaky 7e1e86608e0d7df89c2add23f41140436cf56f58
💔 Build #225504 failed 72e1ac9e74518c53e7d9bd8d83a932260a46588e
💔 Build #225491 failed 315ee82ca73b7040ec0f77d12650359b94d2f36f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

flash1293

Onboarding changes LGTM

PhilippeOberti

LGTM for the Threat Hunting Investigations team

xcrzx self-assigned this Jul 30, 2024

xcrzx changed the title ~~[Security SOlution] Added an endpoint for bootstrapping prebuilt detection rules~~ [Security Solution] Detection rules bootstrap endpoint Jul 30, 2024

elastic deleted a comment from kibana-ci Jul 30, 2024

elastic deleted a comment from kibana-ci Aug 1, 2024

elastic deleted a comment from kibana-ci Aug 2, 2024

xcrzx marked this pull request as ready for review August 2, 2024 10:54

xcrzx requested review from a team as code owners August 2, 2024 10:54

xcrzx requested a review from a team August 2, 2024 10:54

xcrzx requested a review from a team as a code owner August 2, 2024 10:54

xcrzx requested a review from dplumlee August 2, 2024 10:54

botelastic bot added the ci:project-deploy-observability Create an Observability project label Aug 2, 2024

botelastic bot added Team:Fleet Team label for Observability Data Collection Fleet team Team:actionable-obs Formerly "obs-ux-management", responsible for SLO, o11y alerting, significant events, & synthetics. labels Aug 2, 2024

xcrzx requested review from banderror and jpdjere and removed request for dplumlee August 2, 2024 11:09

juliaElastic approved these changes Aug 2, 2024

View reviewed changes

jpdjere reviewed Aug 2, 2024

View reviewed changes

tonyghiani approved these changes Aug 5, 2024

View reviewed changes

maryam-saeidi reviewed Aug 5, 2024

View reviewed changes

jpdjere reviewed Aug 5, 2024

View reviewed changes

maryam-saeidi approved these changes Aug 5, 2024

View reviewed changes

jpdjere approved these changes Aug 5, 2024

View reviewed changes

xcrzx removed the ci:project-deploy-observability Create an Observability project label Aug 6, 2024

Added an endpoint for bootstrapping prebuilt detection rules

e66c663

botelastic bot added the ci:project-deploy-observability Create an Observability project label Aug 7, 2024

flash1293 approved these changes Aug 7, 2024

View reviewed changes

PhilippeOberti approved these changes Aug 7, 2024

View reviewed changes

xcrzx merged commit 6a3c98d into elastic:main Aug 7, 2024

kibanamachine added the backport:skip This PR does not require backporting label Aug 7, 2024

xcrzx deleted the bootstrap-endpoint branch August 7, 2024 12:46

Conversation

xcrzx commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

xcrzx commented Aug 2, 2024

Uh oh!

xcrzx commented Aug 2, 2024

Uh oh!

elasticmachine commented Aug 2, 2024

Uh oh!

elasticmachine commented Aug 2, 2024

Uh oh!

elasticmachine commented Aug 2, 2024

Uh oh!

elasticmachine commented Aug 2, 2024

Uh oh!

elasticmachine commented Aug 2, 2024

Uh oh!

juliaElastic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tonyghiani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maryam-saeidi left a comment

Choose a reason for hiding this comment

Uh oh!

jpdjere left a comment

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Aug 7, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

API count

History

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

PhilippeOberti left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

xcrzx commented Jul 30, 2024 •

edited

Loading