[Cloud Security] Added filter support to graph API by kfirpeled · Pull Request #199048 · elastic/kibana

kfirpeled · 2024-11-05T19:41:40Z

Summary

Enhances the graph API to support filtering by bool query.

Graph API is an internal API that hasn't been released yet to ESS, and is not available yet on serverless (behind a feature-flag in kibana.config) due to the above I don't consider it a breaking change.

Previous API request body:

query: schema.object({
    actorIds: schema.arrayOf(schema.string()),
    eventIds: schema.arrayOf(schema.string()),
    // TODO: use zod for range validation instead of config schema
    start: schema.oneOf([schema.number(), schema.string()]),
    end: schema.oneOf([schema.number(), schema.string()]),

New API request body:

  nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes)
  showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id
  query: schema.object({
    eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red
    // TODO: use zod for range validation instead of config schema
    start: schema.oneOf([schema.number(), schema.string()]),
    end: schema.oneOf([schema.number(), schema.string()]),
    esQuery: schema.maybe( // elasticsearch's dsl bool query
      schema.object({
        bool: schema.object({
          filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
          must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
          should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
          must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
        }),
      })

New field to the graph API response (pseudo):

messages?: ApiMessageCode[]

enum ApiMessageCode {
  ReachedNodesLimit = 'REACHED_NODES_LIMIT',
}

How to test

Toggle feature flag in kibana.dev.yml

xpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled']

To test through the UI you can use the mocked data

node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \ 
  --es-url http://elastic:changeme@localhost:9200 \
  --kibana-url http://elastic:changeme@localhost:5601

node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \
  --es-url http://elastic:changeme@localhost:9200 \
  --kibana-url http://elastic:changeme@localhost:5601

Go to the alerts page
Change the query time range to show alerts from the 13th of October 2024 (IMPORTANT)
Open the alerts flyout
Scroll to see the graph visualization : D

To test only the API you can use the mocked data

node scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \ 
--es-url http://elastic:changeme@localhost:9200 \
--kibana-url http://elastic:changeme@localhost:5601

And through dev tools:

POST kbn:/internal/cloud_security_posture/graph?apiVersion=1
{
  "query": {
    "eventIds": [],
    "start": "now-1y/y",
    "end": "now/d",
    "esQuery": {
      "bool": {
        "filter": [
        {
          "match_phrase": {
            "actor.entity.id": "admin@example.com"
          }
        }
        ]
      }
    }
  }
}

Related PRs

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Graph API is an internal API that hasn't been released yet to ESS, and is not available yet on serverless (behind a feature-flag in kibana.config) due to the above I don't consider it as a breaking change

opauloh

The code looks good, however, when testing the UI I'm seeing a blank canvas instead of the chart.

Could it be because the actor is missing in the esQuery on the payload?

Here's the empty response, also should we think about handling empty response from the chart to not render it for example?

opauloh · 2024-11-07T23:51:40Z

x-pack/test_serverless/api_integration/test_suites/security/cloud_security_posture/graph.ts

  let supertestViewer: Pick<Agent, 'post'>;

-  const postGraph = (agent: Pick<Agent, 'post'>, body: any) => {
+  const postGraph = (agent: Pick<Agent, 'post'>, body: GraphRequest) => {


opauloh · 2024-11-08T00:16:30Z

x-pack/plugins/cloud_security_posture/server/routes/graph/v1.ts


-const parseRecords = (logger: Logger, records: GraphEdge[]): GraphContext => {
-  const ctx: ParseContext = { nodesMap: {}, edgeLabelsNodes: {}, edgesMap: {}, labelEdges: {} };
+const parseRecords = (logger: Logger, records: GraphEdge[], nodesLimit?: number): GraphContext => {


should the parseRecords also ensure values that we expect to be array are parsed?

the docs suggests that string might be returned from VALUES() when there's a single value, that could cause error when accessing the array.includes method below, i.e ips.includes(actorId)

good point,
I'll add tests for these edge cases

opauloh · 2024-11-08T00:19:01Z

x-pack/packages/kbn-cloud-security-posture/common/schema/graph/v1.ts

+      schema.object({
+        bool: schema.object({
+          filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
+          must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
+          should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
+          must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
+        }),
+      })


I'm surprised there wasn't a schema available for esQuery on @kbn/es-query

likewise, that was the best example I found so far so I used it

kfirpeled · 2024-11-08T15:24:07Z

The code looks good, however, when testing the UI I'm seeing a blank canvas instead of the chart.

Could it be because the actor is missing in the esQuery on the payload?

Here's the empty response, also should we think about handling empty response from the chart to not render it for example?

Thanks for the review @opauloh

This is due to the fact I changed the range of time to be relative to the alerts time, however the mocked data is not aligned.

Maybe I can check for the original_event time. Ill fix it on the other PR

…-fix'

PhilippeOberti

left a couple of minor comments, but overall the Threat Hunting Investigations code LGTM

PhilippeOberti · 2024-11-11T15:04:42Z

...plugins/security_solution/public/flyout/document_details/right/hooks/use_fetch_graph_data.ts

-  const { actorIds, eventIds, start, end } = req.query;
+  const { eventIds, start, end, esQuery } = req.query;
  const http = useHttp();
+  const QUERY_KEY = ['useFetchGraphData', eventIds, start, end, esQuery];


should we use useMemo here to avoid creating a new array everytime?

Hey @PhilippeOberti, thanks for bringing this up! I've been thinking about the use of useMemo for the react-query key in this context.

From what I've gathered, React Query creates a hash of query keys internally, in a way that even when creating a new array every time, as long as the contents are the same, React Query will recognize it as the same query and avoid unnecessary refetches.

So I thought that using useMemo would not provide a benefit while adding a bit of complexity to the code. But I'm curious, have you noticed any performance improvements or other benefits from memoizing the query key with useMemo in your experience? I'd like to hear your thoughts on this!

that is a really good point! I think it has become a reflex now when seeing variables like these that I always think about useMemo but in this case I agree it is not necessary or even useful at all!

PhilippeOberti · 2024-11-11T15:05:22Z

...plugins/security_solution/public/flyout/document_details/right/hooks/use_fetch_graph_data.ts

I know it's not part of this PR, but it would be nice to have a couple of unit tests for this hook. Maybe they can be added as part of a future PR?

@PhilippeOberti added :)

kibanamachine · 2024-11-11T19:47:42Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11785186607

elasticmachine · 2024-11-11T19:48:17Z

💚 Build Succeeded

Buildkite Build
Commit: d914dd9

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	13.4MB	13.4MB	+356.0B

History

💛 Build #249977 was flaky fc3dbe6
💔 Build #249972 failed a9c0614
💔 Build #249963 failed 205d97d
💛 Build #249822 was flaky f3813c8
💔 Build #249269 failed b6ddab9
💚 Build #248622 succeeded 8fec92b

## Summary Enhances the graph API to support filtering by bool query. Graph API is an internal API that hasn't been released yet to ESS, and is not available yet on serverless (behind a feature-flag in kibana.config) due to the above I don't consider it a breaking change. Previous API request body: ```js query: schema.object({ actorIds: schema.arrayOf(schema.string()), eventIds: schema.arrayOf(schema.string()), // TODO: use zod for range validation instead of config schema start: schema.oneOf([schema.number(), schema.string()]), end: schema.oneOf([schema.number(), schema.string()]), ``` New API request body: ```js nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes) showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id query: schema.object({ eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red // TODO: use zod for range validation instead of config schema start: schema.oneOf([schema.number(), schema.string()]), end: schema.oneOf([schema.number(), schema.string()]), esQuery: schema.maybe( // elasticsearch's dsl bool query schema.object({ bool: schema.object({ filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), }), }) ``` New field to the graph API response (pseudo): ```js messages?: ApiMessageCode[] enum ApiMessageCode { ReachedNodesLimit = 'REACHED_NODES_LIMIT', } ``` ### How to test Toggle feature flag in kibana.dev.yml ```yaml xpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled'] ``` To test through the UI you can use the mocked data ```bash node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 ``` 1. Go to the alerts page 2. Change the query time range to show alerts from the 13th of October 2024 (**IMPORTANT**) 3. Open the alerts flyout 5. Scroll to see the graph visualization : D To test **only** the API you can use the mocked data ```bash node scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 ``` And through dev tools: ``` POST kbn:/internal/cloud_security_posture/graph?apiVersion=1 { "query": { "eventIds": [], "start": "now-1y/y", "end": "now/d", "esQuery": { "bool": { "filter": [ { "match_phrase": { "actor.entity.id": "admin@example.com" } } ] } } } } ``` ### Related PRs - elastic#196034 - elastic#195307 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 160e626)

kibanamachine · 2024-11-11T19:52:24Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-11-12T19:48:56Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

…199702) # Backport This will backport the following commits from `main` to `8.x`: - [[Cloud Security] Added filter support to graph API (#199048)](#199048)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Kfir Peled <61654899+kfirpeled@users.noreply.github.com>

## Summary Enhances the graph API to support filtering by bool query. Graph API is an internal API that hasn't been released yet to ESS, and is not available yet on serverless (behind a feature-flag in kibana.config) due to the above I don't consider it a breaking change. Previous API request body: ```js query: schema.object({ actorIds: schema.arrayOf(schema.string()), eventIds: schema.arrayOf(schema.string()), // TODO: use zod for range validation instead of config schema start: schema.oneOf([schema.number(), schema.string()]), end: schema.oneOf([schema.number(), schema.string()]), ``` New API request body: ```js nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes) showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id query: schema.object({ eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red // TODO: use zod for range validation instead of config schema start: schema.oneOf([schema.number(), schema.string()]), end: schema.oneOf([schema.number(), schema.string()]), esQuery: schema.maybe( // elasticsearch's dsl bool query schema.object({ bool: schema.object({ filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))), }), }) ``` New field to the graph API response (pseudo): ```js messages?: ApiMessageCode[] enum ApiMessageCode { ReachedNodesLimit = 'REACHED_NODES_LIMIT', } ``` ### How to test Toggle feature flag in kibana.dev.yml ```yaml xpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled'] ``` To test through the UI you can use the mocked data ```bash node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 ``` 1. Go to the alerts page 2. Change the query time range to show alerts from the 13th of October 2024 (**IMPORTANT**) 3. Open the alerts flyout 5. Scroll to see the graph visualization : D To test **only** the API you can use the mocked data ```bash node scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \ --es-url http://elastic:changeme@localhost:9200 \ --kibana-url http://elastic:changeme@localhost:5601 ``` And through dev tools: ``` POST kbn:/internal/cloud_security_posture/graph?apiVersion=1 { "query": { "eventIds": [], "start": "now-1y/y", "end": "now/d", "esQuery": { "bool": { "filter": [ { "match_phrase": { "actor.entity.id": "admin@example.com" } } ] } } } } ``` ### Related PRs - elastic#196034 - elastic#195307 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Added filter support to graph API

1ad3291

Graph API is an internal API that hasn't been released yet to ESS, and is not available yet on serverless (behind a feature-flag in kibana.config) due to the above I don't consider it as a breaking change

kfirpeled added backport:prev-minor release_note:skip Skip the PR/issue when compiling release notes labels Nov 5, 2024

kfirpeled marked this pull request as ready for review November 5, 2024 19:48

kfirpeled requested review from a team as code owners November 5, 2024 19:48

kfirpeled requested a review from opauloh November 5, 2024 19:48

kibanamachine and others added 4 commits November 5, 2024 19:59

[CI] Auto-commit changed files from 'node scripts/notice'

8fec92b

Hide unknown targets by default

18df3a7

Added nodes limit parameter to the API

b6ddab9

Merge branch 'main' into cspm/cdr-graph-api-enhance-query

7c770af

opauloh reviewed Nov 8, 2024

View reviewed changes

opauloh approved these changes Nov 8, 2024

View reviewed changes

kfirpeled and others added 5 commits November 10, 2024 20:06

type fixes

f3813c8

Added messages array for the graph api

869cd13

[CI] Auto-commit changed files from 'node scripts/notice'

205d97d

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

a9c0614

…-fix'

[CI] Auto-commit changed files from 'node scripts/notice'

fc3dbe6

PhilippeOberti approved these changes Nov 11, 2024

View reviewed changes

kfirpeled and others added 2 commits November 11, 2024 17:56

Added UT and memorise query key

d6135fe

Merge branch 'main' into cspm/cdr-graph-api-enhance-query

d914dd9

kfirpeled enabled auto-merge (squash) November 11, 2024 17:59

kfirpeled merged commit 160e626 into elastic:main Nov 11, 2024

kibanamachine added the v9.0.0 label Nov 11, 2024

kibanamachine mentioned this pull request Nov 11, 2024

[8.x] [Cloud Security] Added filter support to graph API (#199048) #199702

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 12, 2024

kibanamachine added v8.17.0 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Nov 13, 2024

Conversation

kfirpeled commented Nov 5, 2024 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Related PRs

Checklist

Uh oh!

opauloh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfirpeled commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilippeOberti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Nov 11, 2024

Uh oh!

elasticmachine commented Nov 11, 2024

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Uh oh!

kibanamachine commented Nov 11, 2024

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Nov 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kfirpeled commented Nov 5, 2024 •

edited by kibanamachine

Loading

kfirpeled commented Nov 8, 2024 •

edited

Loading