Skip to content

[SecuritySolutions] Create Entity Store 'entities/list' API#192806

Merged
machadoum merged 13 commits intoelastic:mainfrom
machadoum:siem-ea-9180-api
Sep 19, 2024
Merged

[SecuritySolutions] Create Entity Store 'entities/list' API#192806
machadoum merged 13 commits intoelastic:mainfrom
machadoum:siem-ea-9180-api

Conversation

@machadoum
Copy link
Copy Markdown
Member

@machadoum machadoum commented Sep 13, 2024

This PR introduces the following API routes for listing Entity Store "entities":

List Entities GET /api/entity_store/entities/list

The PR includes the following:

  • The OpenAPI schemas for the route
  • The actual Kibana side endpoint
  • Add searchEntities function to the EntityStoreDataClient

How to test

  1. Add some host/user data
  2. Make sure to add entityStoreEnabled under xpack.securitySolution.enableExperimental in your kibana.dev.yml
  3. In kibana dev tools or your terminal, call the INIT route for either user or host.
  4. You should now see 2 transforms in kibana. Make sure to re-trigger them if needed so they process the documents.
  5. Call the new API, and it should return entities

Implements https://github.com/elastic/security-team/issues/10517

Checklist

@machadoum machadoum force-pushed the siem-ea-9180-api branch 2 times, most recently from ea4953c to 7e39998 Compare September 17, 2024 07:21
add API test

Add data client test
@machadoum machadoum marked this pull request as ready for review September 17, 2024 09:33
@machadoum machadoum requested review from a team as code owners September 17, 2024 09:33
@machadoum machadoum self-assigned this Sep 17, 2024
@machadoum machadoum added release_note:feature Makes this part of the condensed release notes Theme: entity_analytics Feature:Entity Analytics Security Solution Entity Analytics features Team:Entity Analytics Security Entity Analytics Team labels Sep 17, 2024
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

Copy link
Copy Markdown
Contributor

@hop-dev hop-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work 🚀

@machadoum machadoum changed the title Siem ea 9180 api [SecuritySolutions] Create Entity Store 'entities/list' API Sep 17, 2024
@hop-dev hop-dev requested a review from tiansivive September 17, 2024 11:24
paths: {}
components:
schemas:
UserEntityRecord:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have to keep these schemas in sync with the entity definitions metadata fields, right?
I wonder if there's a way to represent that, maybe deduce the names of the fields from the generated zod records?

Just pointing it out for the future, I don't necessarily think we need to include that in this PR

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, unfortunately, we need to keep the API schema and Entity Store schema in sync.
I couldn't figure out a good way to do that automatically. 😞

Copy link
Copy Markdown
Contributor

@tiansivive tiansivive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Good stuff! 🎉

I have a couple minor comments I think are worth some discussion, even if I don't necessarily think they're blocking the PR

Copy link
Copy Markdown
Contributor

@jbudz jbudz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.buildkite/ftr_security_configs.yml

@machadoum machadoum enabled auto-merge (squash) September 17, 2024 12:51
@hop-dev
Copy link
Copy Markdown
Contributor

hop-dev commented Sep 18, 2024

@elasticmachine merge upstream

@hop-dev
Copy link
Copy Markdown
Contributor

hop-dev commented Sep 19, 2024

@elasticmachine merge upstream

@kibana-ci
Copy link
Copy Markdown

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 5803 5804 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 20.4MB 20.4MB +1.3KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @machadoum

@machadoum machadoum merged commit 27f5da4 into elastic:main Sep 19, 2024
@kibanamachine
Copy link
Copy Markdown
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 192806

Questions ?

Please refer to the Backport tool documentation

@machadoum
Copy link
Copy Markdown
Member Author

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

machadoum added a commit to machadoum/kibana that referenced this pull request Sep 23, 2024
…192806)

This PR introduces the following API routes for listing Entity Store
"entities":

<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
List Entities | GET /api/entity_store/entities/list
-- | --
</div></b>

The PR includes the following:
 - The OpenAPI schemas for the route
 - The actual Kibana side endpoint
 - Add searchEntities function to the `EntityStoreDataClient`
 

### How to test

1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Call the new API, and it should return entities 



Implements elastic/security-team#10517

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
machadoum added a commit that referenced this pull request Sep 23, 2024
…92806) (#193562)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[SecuritySolutions] Create Entity Store 'entities/list' API
(#192806)](#192806)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pablo
Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2024-09-19T12:54:53Z","message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","release_note:feature","backport:prev-minor","Theme:
entity_analytics","Feature:Entity Analytics","Team:Entity
Analytics","v8.16.0"],"number":192806,"url":"https://github.com/elastic/kibana/pull/192806","mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192806","number":192806,"mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature:Entity Analytics Security Solution Entity Analytics features release_note:feature Makes this part of the condensed release notes Team:Entity Analytics Security Entity Analytics Team Theme: entity_analytics v8.16.0 v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants