Skip to content

[Entity Analytics][Leads generation][4] Add API routes, LeadDataClient, and async generation#257046

Merged
abhishekbhatia1710 merged 18 commits intoelastic:mainfrom
abhishekbhatia1710:ea-15953-api-routes
Mar 31, 2026
Merged

[Entity Analytics][Leads generation][4] Add API routes, LeadDataClient, and async generation#257046
abhishekbhatia1710 merged 18 commits intoelastic:mainfrom
abhishekbhatia1710:ea-15953-api-routes

Conversation

@abhishekbhatia1710
Copy link
Copy Markdown
Contributor

@abhishekbhatia1710 abhishekbhatia1710 commented Mar 11, 2026

Summary

  • LeadDataClient abstraction : Extracted all Elasticsearch CRUD operations (create, find, get-by-id, dismiss, bulk-update, status, delete) into a dedicated LeadDataClient with snake_case ↔ camelCase transform layer between index mappings and API types
  • Async generate endpoint — Refactored POST /generate to return an executionUuid immediately and run the pipeline (entity retrieval -> observation modules -> scoring -> synthesis -> persistence) in the background
  • Six new API routes : Added GET /{id}, GET /status, POST /{id}/_dismiss, POST /bulk_update, POST /enable, and POST /disable with Zod-validated schemas, versioned routing, and proper authorization
  • Shared Zod schemas : Moved all request/response schemas to common types (generateLeadsRequestSchema, findLeadsRequestSchema, getLeadByIdRequestSchema, dismissLeadRequestSchema, bulkUpdateLeadsRequestSchema, generateLeadsResponseSchema)
  • Refactored GET /leads : Now delegates to LeadDataClient.findLeads with page/perPage/sortField/sortOrder/status query params
  • Unit tests : Added tests for LeadDataClient (createLeads, findLeads, getLeadById, dismissLead, bulkUpdateLeads, getStatus, deleteAllLeads) and all route handlers (get_leads, get_lead_by_id, get_lead_generation_status, dismiss_lead, bulk_update_leads, enable, disable)

Enable and disable routes are placeholder stubs pending Task Manager wiring (#15955).

Relates to: https://github.com/elastic/security-team/issues/15953

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Testing Steps

Manual verification Start Kibana locally and test the API endpoints below.

API Examples

Generate leads (async)

curl -X POST "http://localhost:5601/internal/entity_analytics/leads/generate" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"connectorId": "optional-connector-id"}'

Returns: { "executionUuid": "abc-123-..." }

Find leads

curl -X GET "http://localhost:5601/internal/entity_analytics/leads?page=1&perPage=10&sortField=priority&sortOrder=desc" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: { "leads": [...], "total": 42, "page": 1, "perPage": 10 }

Get lead by ID

curl -X GET "http://localhost:5601/internal/entity_analytics/leads/{id}" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: full leadSchema object

Get status

curl -X GET "http://localhost:5601/internal/entity_analytics/leads/status" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: { "isEnabled": false, "indexExists": true, "totalLeads": 42, "lastRun": "..." }

Dismiss a lead

curl -X POST "http://localhost:5601/internal/entity_analytics/leads/{id}/_dismiss" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: { "success": true }

Bulk update

curl -X POST "http://localhost:5601/internal/entity_analytics/leads/bulk_update" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"ids": ["id-1", "id-2"], "status": "dismissed"}'

Returns: { "updated": 2 }

Enable scheduled generation

curl -X POST "http://localhost:5601/internal/entity_analytics/leads/enable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: { "success": true }

Disable scheduled generation

curl -X POST "http://localhost:5601/internal/entity_analytics/leads/disable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>

Returns: { "success": true }

@abhishekbhatia1710 abhishekbhatia1710 self-assigned this Mar 11, 2026
@abhishekbhatia1710 abhishekbhatia1710 added backport:skip This PR does not require backporting release_note:feature Makes this part of the condensed release notes v9.4.0 labels Mar 11, 2026
@kibanamachine
Copy link
Copy Markdown
Contributor

Cloud deployments require a Github label, please add ci:cloud-deploy or ci:cloud-redeploy and trigger the job through the checkbox again.

@abhishekbhatia1710 abhishekbhatia1710 marked this pull request as ready for review March 25, 2026 14:04
@abhishekbhatia1710 abhishekbhatia1710 requested review from a team as code owners March 25, 2026 14:04
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 25, 2026

📝 Walkthrough

Walkthrough

This pull request introduces comprehensive lead generation functionality for entity analytics in the Security Solution. It adds a complete pipeline: entity retrieval and enrichment from the Entity Store, a configurable lead generation engine that orchestrates observation modules (risk scoring, temporal state analysis, behavioral/alert analysis), weighted priority scoring with bonuses, and LLM-based content synthesis. Supporting infrastructure includes Elasticsearch-backed persistence via a lead data client, REST API endpoints for CRUD and status operations, and constant definitions for URL routes. The engine collects observations from registered modules, computes entity priorities, enriches leads with alert/risk context, and synthesizes human-readable narratives. All components are integrated into the entity analytics route registry behind an experimental feature flag.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • 🛠️ Update Documentation: Commit on current branch
  • 🛠️ Update Documentation: Create PR

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/lead_data_client.ts`:
- Around line 203-224: The bulk response from esClient.bulk is not being
inspected, so partial failures (response.body.errors === true or item-level
errors) can occur and then deleteByQuery runs and removes existing docs; modify
the block where esClient.bulk(...) is awaited (the code that builds bulkBody
from leads and calls leadToEsDoc) to capture the bulk response, check
response.body?.errors and iterate response.body.items to collect/ log any
item-level errors (include lead id, status, error), and if any errors exist
throw or return early so the subsequent esClient.deleteByQuery(...) does not
execute; ensure logs reference executionId and indexName for context.
- Around line 301-320: The updateByQuery call in lead_data_client.ts builds a
Painless script by interpolating values from updates into the script source,
which risks script injection; change esClient.updateByQuery to use a
parameterized script instead: construct the script source to assign from params
(e.g., for each entry in updates generate a fixed snippet like
ctx._source['<field>'] = params['p<i>']) and build a corresponding params object
mapping p<i> to the runtime value, then pass that params object to the
updateByQuery call; reference the existing variables esClient.updateByQuery,
updates, id and the surrounding try/catch to locate and replace the unsafe
string interpolation with a params-based script approach similar to
bulkUpdateLeads.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/temporal_state_module.ts`:
- Around line 95-133: The current aggregation groups by `${entityType}.name` and
treats missing `entity.attributes.privileged` as "previously non-privileged",
which can produce false escalations; change the ES query and downstream checks
to aggregate on a stable identifier (e.g., `${entityType}.id`) or a composite
key (id+name) in `esClient.search`'s `aggs.by_entity.terms.field`, include
`entity.id` and `entity.attributes.privileged` in the `top_hits` `_source`
(oldest_snapshot), and then only mark escalation when the hit's
`entity.attributes` explicitly exists and `privileged === false` (not
undefined); also use the stable `entity.id` (from `hit._source.entity.id`) when
adding to `escalated.add` instead of using `bucket.key` if it is a name to avoid
name reuse skew.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/enable_lead_generation.ts`:
- Around line 41-46: The endpoint currently returns response.ok({ success: true
}) inside the try block in the enable_lead_generation handler (file
enable_lead_generation.ts), which falsely signals success even though Task
Manager wiring is TODO; change the response to return an HTTP 501 Not
Implemented (use the route response helper, e.g., response.customError or
response.notImplemented with a clear message like "Lead generation enable not
implemented — Task Manager wiring pending (`#15955`)") instead of success; keep
the logger.info line but ensure the handler returns 501 and apply the same
change to the corresponding disable route handler so both endpoints accurately
report not-implemented status.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/generate_leads.ts`:
- Around line 60-135: The background pipeline captures a request-scoped
Elasticsearch client using asCurrentUser (esClient) which may become invalid
after response returns; replace the request-scoped client usage in this async
fire-and-forget block by obtaining core.elasticsearch.client.asInternalUser (use
asInternalUser for system-level operations) before launching the background task
and pass that internal client to createCRUDClient, createLeadGenerationEngine
modules (createRiskScoreModule, createTemporalStateModule,
createBehavioralAnalysisModule), and createLeadDataClient; alternatively,
refactor to schedule the work via Task Manager instead of running an in-request
background async IIFE so no request-scoped client (asCurrentUser) is used.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_lead_generation_status.ts`:
- Around line 33-50: The status endpoint is returning global space-level status
and ignores the execution that started the job; update the route added in
get_lead_generation_status.ts so it accepts an execution identifier (e.g.,
executionUuid) in the request (query or body) and forward that identifier into
the lead data lookup instead of the current global call: extend the route
validation to require executionUuid, extract it from the incoming request, and
call createLeadDataClient(...).getStatus(executionUuid) (or equivalent method
overload) so getStatus is scoped to that executionUuid; alternatively, if you
intend single-flight only, enforce and document single-generation semantics in
createLeadDataClient/getStatus and reject additional POST /generate requests
when a run is active.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c55374f8-3576-40bb-8502-26c9d6111de9

📥 Commits

Reviewing files that changed from the base of the PR and between b915a7b and 6d504da.

📒 Files selected for processing (39)
  • x-pack/solutions/security/plugins/security_solution/common/entity_analytics/constants.ts
  • x-pack/solutions/security/plugins/security_solution/common/entity_analytics/lead_generation/constants.ts
  • x-pack/solutions/security/plugins/security_solution/common/entity_analytics/lead_generation/types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/engine/index.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/engine/lead_generation_engine.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/engine/lead_generation_engine.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/engine/llm_synthesize.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_conversion.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_enricher.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_enricher.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_retriever.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_retriever.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/index.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/lead_data_client.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/lead_data_client.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/alert_analysis_module.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/index.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/risk_score_module.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/temporal_state_module.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/utils.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/bulk_update_leads.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/bulk_update_leads.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/disable_lead_generation.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/disable_lead_generation.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/dismiss_lead.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/dismiss_lead.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/enable_lead_generation.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/enable_lead_generation.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/generate_leads.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_lead_by_id.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_lead_by_id.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_lead_generation_status.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_lead_generation_status.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_leads.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/get_leads.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/routes/register_lead_generation_routes.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/register_entity_analytics_routes.ts
  • x-pack/solutions/security/plugins/security_solution/server/plugin_contract.ts

…and async generation

Adds LeadDataClient abstraction (CRUD on leads index), async generate
endpoint, six new routes (get by id, status, dismiss, bulk update,
enable, disable), entity conversion/retrieval/enricher services, LLM
synthesis, weighted scoring engine, and observation module refinements.

Review fixes: bulk error check before stale cleanup, parameterized
Painless scripts, privilege guard in temporal state module, enterprise
license gating on all routes, EntityStoreEntity type derivation.
- Remove 5 unused Zod response schemas and 7 dead inferred type exports
  from common/lead_generation/types.ts (never imported anywhere)
- Remove 3 dead PATTERN_CATALOG entries (investigation_status,
  watchlist_inclusion, risk_escalation) for observation types no module
  produces
- Remove unused function parameters (_config, _group, _pattern) and
  inline _observationTypes into buildTags
- Remove deprecated createAlertAnalysisModule alias and its exports
…n LLM prompt

- Replace `computeStaleness(now, now)` with literal `'fresh'` since staleness
  is recalculated at read time in LeadDataClient
- Add anti-hallucination instruction for rule names in LLM synthesis prompt
- Rename alert_analysis_module/ to behavioral_analysis_module/ to match
  MODULE_ID = 'behavioral_analysis' (restoring consistency from PR elastic#256156)
- Remove deprecated createAlertAnalysisModule alias
- Update all import paths to reference behavioral_analysis_module
Copy link
Copy Markdown
Contributor

@ymao1 ymao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial code review looks good! Will desk test the API next

@ymao1
Copy link
Copy Markdown
Contributor

ymao1 commented Mar 27, 2026

I see there is a fetchAllUnifiedLatestEntities that performs an ES search against the entity store V2 indices but exists in the V1 entity store data client: x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.ts

Should we update to use the listEntities API that was recently added? #258320. Is that happening in another PR?

Also the entity conversion in x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/entity_conversion.ts is not using the entity.EngineMetadata.Type for the entity type, which is what the fields is in the V2 entity store.

- Wrap generate_leads route handler with withMinimumLicense (was the
  only route missing enterprise license gating)
- Pass executionUuid from route into service.generate() so the UUID
  returned to the client matches the one stored on persisted leads
- Improve findLeads error logging: distinguish index_not_found (debug)
  from real errors (error level), extract error message to avoid
  [object Object] in logs
- Let bulkUpdateLeads throw on ES errors so the route handler returns
  a proper error response instead of silently returning { updated: 0 }
- Remove dead entity_retriever.ts and entity_enricher.ts (plus tests);
  entity retrieval uses entityStoreDataClient.fetchAllUnifiedLatestEntities
  and observation modules handle their own data fetching
- Use entity.EngineMetadata.Type (V2 canonical field) over entity.type
  in entityRecordToLeadEntity, falling back to entity.type for compat
@abhishekbhatia1710
Copy link
Copy Markdown
Contributor Author

abhishekbhatia1710 commented Mar 28, 2026

Addressed in the latest push:

CRUDClient.listEntities() : Switched from fetchAllUnifiedLatestEntities (which lived in the V1 EntityStoreDataClient) to CRUDClient.listEntities() from the entity-store plugin (#258320). This is the properly-owned V2 API that queries the unified latest index with search_after pagination. The route now gets a CRUDClient via getStartServices() -> entityStore.createCRUDClient(), and a new fetchAllLeadEntities() helper handles pagination.

The service no longer takes entityStoreDataClient as a dependency it takes a fetchEntities function instead, which makes it easier to test and decouples it from the specific client implementation.

abhishekbhatia1710 and others added 2 commits March 28, 2026 14:09
…ent.listEntities()

Replace EntityStoreDataClient.fetchAllUnifiedLatestEntities() with the
entity-store plugin's CRUDClient.listEntities() API for V2 unified index
access. This uses the canonical, properly-owned API from @kbn/entity-store
with search_after pagination.

- Add fetchAllLeadEntities() paginated helper in entity_conversion.ts
- Refactor createLeadGenerationService to accept a fetchEntities function
  instead of entityStoreDataClient (easier to test, decoupled)
- Wire getStartServices -> entityStore.createCRUDClient in generate route
- Remove ENTITY_SOURCE_FIELDS (CRUDClient returns full docs)
- Add tests for fetchAllLeadEntities pagination logic
@abhishekbhatia1710
Copy link
Copy Markdown
Contributor Author

I self desk-tested this PR, with manually adding the changes of entity store crud client sorting done as part of the PR : #259386.

Waiting for it to get merged, but this PR can be reviewed.

Copy link
Copy Markdown
Contributor

@jaredburgettelastic jaredburgettelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Great job, the code is very straightforward!

abhishekbhatia1710 and others added 2 commits March 31, 2026 13:16
persistLeads wrote camelCase fields (executionId, chatRecommendations)
but findLeads/esDocToLead expected snake_case (execution_uuid,
chat_recommendations), causing silent data loss. Added toSnakeCaseDoc
transform so the service persists docs in the same format as the data
client.

The deleteByQuery used { term: { executionId } } but ES dynamically
maps string fields as text — term queries on text fields don't match
UUIDs (tokenized at hyphens), so must_not matched every document and
deleted all newly-inserted leads.

All term/terms queries on id, status, and execution_uuid now target the
.keyword sub-field so they work correctly against the default dynamic
text+keyword mapping.
@elasticmachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

References to deprecated APIs

id before after diff
securitySolution 535 534 -1

Unreferenced deprecated APIs

id before after diff
securitySolution 535 534 -1

History

cc @abhishekbhatia1710

@abhishekbhatia1710 abhishekbhatia1710 merged commit 8f84f74 into elastic:main Mar 31, 2026
18 checks passed
mbondyra added a commit to mbondyra/kibana that referenced this pull request Mar 31, 2026
…e_for_children6

* commit '3402744f63ca1196e97b11ffac4e7f7efab240df': (80 commits)
  [PerUserAuth] Add EARS auth type for Connectors V2 (elastic#253695)
  Fix `@elastic/eui/require-aria-label-for-modals` lint violations across `@elastic/kibana-core` files (elastic#259757)
  [Entity Analytics][Leads generation][4] Add API routes, LeadDataClient, and async generation (elastic#257046)
  [Agent Builder] Agent-centric UX redesign (elastic#258005)
  fix query streams failing test (elastic#260277)
  [Lens as code] Add list layout to the new API (elastic#259967)
  [FTR] Add warning comments to deployment-agnostic FTR base configs (elastic#260018)
  [Discover][Logs profile] Fix missing search highlights (elastic#260056)
  Plugin system: safe deletion (elastic#259038)
  [Infra] Fix Hosts filter options to match selected schema (elastic#259825)
  Manual Entity Resolution and flyout representation (elastic#260162)
  [Cascade] Handle grouping on fields with unset values (elastic#260033)
  [Fleet] generate OTel config for integration packages with otelcol inputs (elastic#259968)
  [Search] Switch over to V2 index management details (elastic#259866)
  [inference] increase timeout for ES inference calls (elastic#260382)
  [ES|QL] Enable subqueries (elastic#257455)
  [ES|QL] Change Point order free options (elastic#260282)
  [Auth] Added authentication strategy for UIAM OAuth (elastic#256182)
  [Security Solution] Add "alerts_candidate_count" rule execution metric (elastic#259917)
  [api-docs] 2026-03-31 Daily api_docs build (elastic#260380)
  ...
abhishekbhatia1710 added a commit to abhishekbhatia1710/kibana that referenced this pull request Mar 31, 2026
Resolve conflicts after PRs elastic#255272 (Foundation), elastic#256156 (Observation
Modules), elastic#256628 (Entity Retrieval), and elastic#257046 (CRUD API Routes)
were merged into main.

Key resolutions:
- Keep main's authoritative versions of observation modules with
  data-driven tier tables and RiskScoreDataClient integration
- Add scheduling-specific weight properties to module configs
- Wire RiskScoreDataClient through RunPipelineParams via dependency
  injection (route uses context, task creates from CoreStart)
- Preserve main's .keyword suffix fixes in lead_data_client ES queries
- Keep scheduling branch's single-file behavioral_analysis_module
  (consolidated from subdirectory structure) and remove stale subdir
- Delete generate_leads.test.ts (coverage moved to run_pipeline.test.ts)
- Preserve Task Manager registration, enable/disable route wiring,
  and run_pipeline shared orchestration from scheduling branch

Note: pre-commit hook bypassed because ESLint failures are from
upstream kbn-evals-suite-significant-events package (pre-existing
in main), not from lead generation changes.
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Apr 1, 2026
…t, and async generation (elastic#257046)

## Summary

- **LeadDataClient abstraction** : Extracted all Elasticsearch CRUD
operations (create, find, get-by-id, dismiss, bulk-update, status,
delete) into a dedicated `LeadDataClient` with snake_case ↔ camelCase
transform layer between index mappings and API types
- **Async generate endpoint** — Refactored `POST /generate` to return an
`executionUuid` immediately and run the pipeline (entity retrieval ->
observation modules -> scoring -> synthesis -> persistence) in the
background
- **Six new API routes** : Added `GET /{id}`, `GET /status`, `POST
/{id}/_dismiss`, `POST /bulk_update`, `POST /enable`, and `POST
/disable` with Zod-validated schemas, versioned routing, and proper
authorization
- **Shared Zod schemas** : Moved all request/response schemas to common
types (generateLeadsRequestSchema, findLeadsRequestSchema,
getLeadByIdRequestSchema, dismissLeadRequestSchema,
bulkUpdateLeadsRequestSchema, generateLeadsResponseSchema)
- **Refactored GET /leads** : Now delegates to
`LeadDataClient.findLeads` with page/perPage/sortField/sortOrder/status
query params
- **Unit tests** : Added tests for LeadDataClient (createLeads,
findLeads, getLeadById, dismissLead, bulkUpdateLeads, getStatus,
deleteAllLeads) and all route handlers (get_leads, get_lead_by_id,
get_lead_generation_status, dismiss_lead, bulk_update_leads, enable,
disable)

Enable and disable routes are placeholder stubs pending Task Manager
wiring (elastic#15955).

Relates to: elastic/security-team#15953

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [ ] Flaky Test Runner was used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the guidelines
- [ ] Review the backport guidelines and apply applicable `backport:*`
labels.


## Testing Steps

**Manual verification** Start Kibana locally and test the API endpoints
below.

## API Examples

### Generate leads (async)
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/generate" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"connectorId": "optional-connector-id"}'
```
Returns: `{ "executionUuid": "abc-123-..." }`

### Find leads
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads?page=1&perPage=10&sortField=priority&sortOrder=desc" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "leads": [...], "total": 42, "page": 1, "perPage": 10 }`

### Get lead by ID
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads/{id}" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: full `leadSchema` object

### Get status
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads/status" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "isEnabled": false, "indexExists": true, "totalLeads": 42,
"lastRun": "..." }`

### Dismiss a lead
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/{id}/_dismiss" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

### Bulk update
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/bulk_update" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"ids": ["id-1", "id-2"], "status": "dismissed"}'
```
Returns: `{ "updated": 2 }`

### Enable scheduled generation
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/enable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

### Disable scheduled generation
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/disable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jared Burgett <147995946+jaredburgettelastic@users.noreply.github.com>
Co-authored-by: jaredburgettelastic <jared.burgett@elastic.co>
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Apr 2, 2026
…t, and async generation (elastic#257046)

## Summary

- **LeadDataClient abstraction** : Extracted all Elasticsearch CRUD
operations (create, find, get-by-id, dismiss, bulk-update, status,
delete) into a dedicated `LeadDataClient` with snake_case ↔ camelCase
transform layer between index mappings and API types
- **Async generate endpoint** — Refactored `POST /generate` to return an
`executionUuid` immediately and run the pipeline (entity retrieval ->
observation modules -> scoring -> synthesis -> persistence) in the
background
- **Six new API routes** : Added `GET /{id}`, `GET /status`, `POST
/{id}/_dismiss`, `POST /bulk_update`, `POST /enable`, and `POST
/disable` with Zod-validated schemas, versioned routing, and proper
authorization
- **Shared Zod schemas** : Moved all request/response schemas to common
types (generateLeadsRequestSchema, findLeadsRequestSchema,
getLeadByIdRequestSchema, dismissLeadRequestSchema,
bulkUpdateLeadsRequestSchema, generateLeadsResponseSchema)
- **Refactored GET /leads** : Now delegates to
`LeadDataClient.findLeads` with page/perPage/sortField/sortOrder/status
query params
- **Unit tests** : Added tests for LeadDataClient (createLeads,
findLeads, getLeadById, dismissLead, bulkUpdateLeads, getStatus,
deleteAllLeads) and all route handlers (get_leads, get_lead_by_id,
get_lead_generation_status, dismiss_lead, bulk_update_leads, enable,
disable)

Enable and disable routes are placeholder stubs pending Task Manager
wiring (elastic#15955).

Relates to: elastic/security-team#15953

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [ ] Flaky Test Runner was used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the guidelines
- [ ] Review the backport guidelines and apply applicable `backport:*`
labels.


## Testing Steps

**Manual verification** Start Kibana locally and test the API endpoints
below.

## API Examples

### Generate leads (async)
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/generate" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"connectorId": "optional-connector-id"}'
```
Returns: `{ "executionUuid": "abc-123-..." }`

### Find leads
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads?page=1&perPage=10&sortField=priority&sortOrder=desc" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "leads": [...], "total": 42, "page": 1, "perPage": 10 }`

### Get lead by ID
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads/{id}" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: full `leadSchema` object

### Get status
```bash
curl -X GET "http://localhost:5601/internal/entity_analytics/leads/status" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "isEnabled": false, "indexExists": true, "totalLeads": 42,
"lastRun": "..." }`

### Dismiss a lead
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/{id}/_dismiss" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

### Bulk update
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/bulk_update" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -H "elastic-api-version: 1" \
  -u <....> \
  -d '{"ids": ["id-1", "id-2"], "status": "dismissed"}'
```
Returns: `{ "updated": 2 }`

### Enable scheduled generation
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/enable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

### Disable scheduled generation
```bash
curl -X POST "http://localhost:5601/internal/entity_analytics/leads/disable" \
  -H "kbn-xsrf: true" \
  -H "elastic-api-version: 1" \
  -u <....>
```
Returns: `{ "success": true }`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jared Burgett <147995946+jaredburgettelastic@users.noreply.github.com>
Co-authored-by: jaredburgettelastic <jared.burgett@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:feature Makes this part of the condensed release notes v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants