[Entity Store] Adding list endpoint with query filter by tiansivive · Pull Request #258320 · elastic/kibana

tiansivive · 2026-03-18T11:05:56Z

Summary

This PR introduces a LIST endpoint in the entity store, filterable via query param

How to test

Seed some data into the store. The easiest way is via security-document-generator.
Test the endpoint with curl

List all entities (no filter)

curl -s -u elastic:changeme \
  -H "kbn-xsrf: true" \
  -H "x-elastic-internal-origin: kibana" \
  -H "elastic-api-version: 2" \
  "http://localhost:5601/internal/security/entity_store/entities" | jq

Expected: { "entities": [ ... ] } containing seeded entities.

List with a DSL filter

curl -s -u elastic:changeme \
  -H "kbn-xsrf: true" \
  -H "x-elastic-internal-origin: kibana" \
  -H "elastic-api-version: 2" \
  --get \
  --data-urlencode 'filter={"term":{"entity.type":"host"}}' \
  "http://localhost:5601/internal/security/entity_store/entities" | jq

Expected: only hosts entities are returned.

Invalid filter (should return 400)

curl -s -u elastic:changeme \
  -H "kbn-xsrf: true" \
  -H "x-elastic-internal-origin: kibana" \
  -H "elastic-api-version: 2" \
  "http://localhost:5601/internal/security/entity_store/entities?filter=not-json" | jq

Expected: 400 Bad Request with "Invalid filter: must be a valid JSON-encoded DSL query".

chennn1990

Looking good,
Please see my comments

chennn1990 · 2026-03-18T12:12:22Z

x-pack/solutions/security/plugins/entity_store/server/routes/apis/crud/get.ts

+import { wrapMiddlewares } from '../../middleware';
+
+const querySchema = z.object({
+  filter: z.string().optional(),


please add a validation for the filter

Actually, shouldnt we be using OAS?

lets keep it internal for Now, once we will convert the CRUD API's to be public this will be addressed
@uri-weisman WDYT?

I mean using the superRefine method in order to validate the filter string., it should be a valid DSL filter.

superRefine is apparently deprecated?
I also didnt really see any other place doing this kind of validation, so isntead I jsut used the typical KQL to query DSL utility fns. They already throw on incorrect strings, so jus tuse that to validate.

chennn1990 · 2026-03-18T12:13:26Z

x-pack/solutions/security/plugins/entity_store/server/routes/apis/crud/list.ts

+          }
+        }
+
+        const entities = await crudClient.listEntities(parsedFilter);


use this one inside the try

chennn1990 · 2026-03-18T12:15:56Z

x-pack/solutions/security/plugins/entity_store/server/domain/crud/crud_client.ts

+  // An optional DSL filter can be provided and is applied as an additional
+  // filter clause on the search query, e.g. to scope results by additional
+  // field conditions.
+  public async listEntities(filter?: QueryDslQueryContainer): Promise<Entity[]> {


I would change the terminology to getEntities.
I see list as a quite confusing inside CRUD operations

the problem with GET is that it implies a single entity? We typically use LIST for GET'ing multiple entries. I can change though, if that's the way we should look at API's in this domain?

We don't have a convention yet, lets stick to the Kibana conventions.
If list is more common, lets keep it

chennn1990 · 2026-03-18T12:17:49Z

x-pack/solutions/security/plugins/entity_store/common/index.ts

  STOP: `${ENTITY_STORE_BASE_ROUTE}/stop`,
  FORCE_LOG_EXTRACTION: `${ENTITY_STORE_BASE_ROUTE}/{entityType}/force_log_extraction`,
  FORCE_HISTORY_SNAPSHOT: `${ENTITY_STORE_BASE_ROUTE}/force_history_snapshot`,
+  CRUD_LIST: `${ENTITY_STORE_BASE_ROUTE}/entities`,


CRUD_GET: ENTITY_STORE_BASE_ROUTE
IMO A simple get to the base route is more meaningful

uri-weisman

@tiansivive thanks for putting this PR 🙏
Can you please add unit tests and a Scout test (At least one happy path)?

uri-weisman · 2026-03-18T12:30:16Z

x-pack/solutions/security/plugins/entity_store/server/domain/crud/crud_client.ts

+      ? { bool: { filter: [filter] } }
+      : { match_all: {} };
+
+    const resp = await this.esClient.search<Entity>({


So today you get at most one page of 10 results (ES default), with no way to request more or skip ahead.
I suggest to extend the params to support size and searchAfter to make this endpoint more usable.

export interface ListEntitiesParams { filter?: QueryDslQueryContainer; size?: number; searchAfter?: Array<string | number>; }

uri-weisman

Good Job @tiansivive!

chennn1990 · 2026-03-22T10:53:11Z

x-pack/solutions/security/plugins/entity_store/test/scout/api/tests/crud_api.spec.ts

@@ -74,6 +74,37 @@ apiTest.describe.skip('Entity Store CRUD API tests', { tag: ENTITY_STORE_TAGS },
    expect(check.found).toBe(true);
  });

+  apiTest('Should list entities', async ({ apiClient, esClient }) => {


This test feels a bit limited, I would expect to see multiple tests that better cover the different API capabilities, especially around ListEntitiesParams (valid scenarios & invalid scenarios)

chennn1990 · 2026-03-22T10:55:56Z

x-pack/solutions/security/plugins/entity_store/server/routes/apis/crud/get.ts

+import { wrapMiddlewares } from '../../middleware';
+
+const querySchema = z.object({
+  filter: z.string().optional(),


I mean using the superRefine method in order to validate the filter string., it should be a valid DSL filter.

elasticmachine · 2026-03-24T13:49:43Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 1fb3b9a

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #160 / Actions and Triggers app Connectors Tines connector page should edit the connector
[job] [logs] FTR Configs #26 / InfraOps App Metrics UI Hosts View #With data #Single Host Flyout Tabs Dashboards Tab should render dashboards tab splash screen with option to add dashboard

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`entityStore`	24.5KB	24.5KB	+26.0B

History

💛 Build #414633 was flaky 12a6c34
💛 Build #412785 was flaky e01f3f3
💚 Build #412149 succeeded d20d9f5

Steps 2b + 2e from refactor prompt: - Replace entityStoreDataClient param with listEntities: () => Promise<LeadEntity[]> in RunPipelineParams (dependency injection pattern) - Wire startPlugins.entityStore.createCRUDClient() + fetchAllLeadEntities() in the task runner, removing the TODO skip - Add fetchAllLeadEntities paginated helper to entity_conversion.ts - Rewrite run_pipeline.test.ts to mock listEntities function Depends on PR elastic#258320 for CRUDClient.listEntities() method.

Steps 2a, 2c, 2d from refactor prompt: - Add entityStore: EntityStoreStartContract to SecuritySolutionPluginStartDependencies - Create fetchAllLeadEntities() paginated helper in entity_conversion.ts that pages through all V2 entities via searchAfter cursors - Use startPlugins.entityStore.createCRUDClient().listEntities() in the generate_leads route instead of EntityStoreDataClient.searchEntities() - Restore getStartServices parameter to generateLeadsRoute (needed for entityStore plugin access, not for inference/chatModel) Depends on PR elastic#258320 for CRUDClient.listEntities() method.

…ra/kibana into dashboard_align_attachment_to_api * 'dashboard_align_attachment_to_api' of github.com:mbondyra/kibana: (45 commits) [OTel Tracing] HTTP instrumentation (elastic#258663) Replace deprecated EUI icons in files owned by @elastic/ml-ui (elastic#255624) [Codeowners] add missing codeowners for security_solution_api_integration tests (elastic#259223) [CI] fix bad imports that came from a merge-race (elastic#259383) Add `.claude/worktrees/` to `.gitignore` (elastic#259192) Improve unknown-key validation error message in @kbn/config-schema (elastic#258633) [ML] Update Security ML jobs to use entity analytics fields for host and user fields (elastic#255339) [Table sweep] Update table columns responsiveness in Index Management and Dashboards (elastic#259340) skip failing test suite (elastic#258790) skip failing test suite (elastic#259261) chore: util to clean cached images (elastic#259335) [Entity Store] Use last_seen for automated resolution watermark (elastic#258574) [One Workflow] Fix flaky alert trigger Scout test by removing order-dependent assertions (elastic#259299) Skip serverless Discover request counts tests for MKI (elastic#259333) [Security Solution] render header title in new document flyout in Security Solution and Discover (elastic#258166) [Agent Builder] register inference endpoint feature (elastic#259259) [Agent Builder] Skills Command Menu - Add descriptions and scope options to agent (elastic#258964) [Streams][Streamlang][API] Fully use meta({id}) to reuse schema partials in OAS output (elastic#259275) fix(files_example): add tableCaption to EuiInMemoryTable for a11y (elastic#258289) [Entity Store] Adding list endpoint with query filter (elastic#258320) ...

Steps 2b + 2e from refactor prompt: - Replace entityStoreDataClient param with listEntities: () => Promise<LeadEntity[]> in RunPipelineParams (dependency injection pattern) - Wire startPlugins.entityStore.createCRUDClient() + fetchAllLeadEntities() in the task runner, removing the TODO skip - Add fetchAllLeadEntities paginated helper to entity_conversion.ts - Rewrite run_pipeline.test.ts to mock listEntities function Depends on PR elastic#258320 for CRUDClient.listEntities() method.

# Summary This PR introduces a LIST endpoint in the entity store, filterable via query param # How to test 1. Seed some data into the store. The easiest way is via [security-document-generator](https://github.com/elastic/security-documents-generator). 2. Test the endpoint with curl ### List all entities (no filter) ```bash curl -s -u elastic:changeme \ -H "kbn-xsrf: true" \ -H "x-elastic-internal-origin: kibana" \ -H "elastic-api-version: 2" \ "http://localhost:5601/internal/security/entity_store/entities" | jq ``` Expected: `{ "entities": [ ... ] }` containing seeded entities. ### List with a DSL filter ```bash curl -s -u elastic:changeme \ -H "kbn-xsrf: true" \ -H "x-elastic-internal-origin: kibana" \ -H "elastic-api-version: 2" \ --get \ --data-urlencode 'filter={"term":{"entity.type":"host"}}' \ "http://localhost:5601/internal/security/entity_store/entities" | jq ``` Expected: only hosts entities are returned. --- ### Invalid filter (should return 400) ```bash curl -s -u elastic:changeme \ -H "kbn-xsrf: true" \ -H "x-elastic-internal-origin: kibana" \ -H "elastic-api-version: 2" \ "http://localhost:5601/internal/security/entity_store/entities?filter=not-json" | jq ``` Expected: `400 Bad Request` with `"Invalid filter: must be a valid JSON-encoded DSL query"`.

tiansivive added Entity Store (Next) release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 18, 2026

tiansivive marked this pull request as ready for review March 18, 2026 11:06

tiansivive requested a review from a team as a code owner March 18, 2026 11:06

chennn1990 reviewed Mar 18, 2026

View reviewed changes

uri-weisman reviewed Mar 18, 2026

View reviewed changes

uri-weisman approved these changes Mar 20, 2026

View reviewed changes

chennn1990 reviewed Mar 22, 2026

View reviewed changes

tiansivive added 4 commits March 23, 2026 14:29

adding list endpoint with query filter

9da5477

wrap in try/catch

2dd9596

addressing review

3e65b84

review

12a6c34

tiansivive force-pushed the siem/entity-store/crud-api-get-filter branch from e01f3f3 to 12a6c34 Compare March 23, 2026 13:30

tiansivive added 2 commits March 23, 2026 17:03

mo tests

6768a41

params validation

2a8e323

chennn1990 approved these changes Mar 24, 2026

View reviewed changes

tiansivive enabled auto-merge (squash) March 24, 2026 12:23

Merge branch 'main' into siem/entity-store/crud-api-get-filter

1fb3b9a

tiansivive merged commit 91ae003 into elastic:main Mar 24, 2026
18 checks passed

kibanamachine added the v9.4.0 label Mar 24, 2026

ymao1 mentioned this pull request Mar 27, 2026

[Entity Analytics][Leads generation][4] Add API routes, LeadDataClient, and async generation #257046

Merged

3 tasks

Conversation

tiansivive commented Mar 18, 2026

Summary

How to test

List all entities (no filter)

List with a DSL filter

Invalid filter (should return 400)

Uh oh!

chennn1990 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uri-weisman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uri-weisman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Mar 24, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Page load bundle

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants