…ed types, index templates, observation module interface (elastic#255272)

## Summary

Adds the foundational layer for the Entity Analytics Lead Generation
feature:

- **Feature flag** (`leadGenerationEnabled`) to gate the entire pipeline
- **Shared types and Zod schemas** for leads, observations, entities,
engine config, and API contracts
- **Elasticsearch index templates** with mappings and lifecycle service
for lead persistence (adhoc + scheduled modes)
- **Observation module interface** and **registry** for pluggable,
priority-ordered, fault-tolerant module execution

This PR contains no runtime wiring, it establishes the contracts and
infrastructure that subsequent PRs will build on.

Relates to: elastic/security-team#15946

## Testing

1. **Unit tests** run the following commands individually:

```
yarn test:jest x-pack/solutions/security/plugins/security_solution/common/entity_analytics/lead_generation/types.test.ts
yarn test:jest x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/indices/lead_index_service.test.ts
yarn test:jest x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/observation_module_registry.test.ts
```

2. **Type check** — verify no type errors:

```
yarn test:type_check --project x-pack/solutions/security/plugins/security_solution/tsconfig.json
```

3. **Feature flag** — confirm `leadGenerationEnabled` defaults to
`false` and does not alter existing behavior when disabled.

…ed types, index templates, observation module interface (elastic#255272)

## Summary

Adds the foundational layer for the Entity Analytics Lead Generation
feature:

- **Feature flag** (`leadGenerationEnabled`) to gate the entire pipeline
- **Shared types and Zod schemas** for leads, observations, entities,
engine config, and API contracts
- **Elasticsearch index templates** with mappings and lifecycle service
for lead persistence (adhoc + scheduled modes)
- **Observation module interface** and **registry** for pluggable,
priority-ordered, fault-tolerant module execution

This PR contains no runtime wiring, it establishes the contracts and
infrastructure that subsequent PRs will build on.

Relates to: elastic/security-team#15946

## Testing

1. **Unit tests** run the following commands individually:

```
yarn test:jest x-pack/solutions/security/plugins/security_solution/common/entity_analytics/lead_generation/types.test.ts
yarn test:jest x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/indices/lead_index_service.test.ts
yarn test:jest x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/lead_generation/observation_modules/observation_module_registry.test.ts
```

2. **Type check** — verify no type errors:

```
yarn test:type_check --project x-pack/solutions/security/plugins/security_solution/tsconfig.json
```

3. **Feature flag** — confirm `leadGenerationEnabled` defaults to
`false` and does not alter existing behavior when disabled.

…behavorial observation modules (#256156)

## Summary

This PR adds three pluggable observation modules to the lead generation
pipeline. Each module collects signals about entities (users and hosts)
from different data sources. The engine combines these observations to
score and rank leads, and uses them when generating titles, tags, and
chat recommendations.

- **Risk Analysis** : Uses current risk scores on entity records and
(where available) risk score history. Flags entities that are high- or
critical-risk, and those with meaningful risk increases over 24 hours, 7
days, or 90 days. Privileged entities with elevated risk are called out
separately.

- **Temporal State Analysis** : Looks at how entity state changes over
time. Right now it focuses on **privilege escalation**: entities that
were not privileged in the past but are privileged in the latest view
(using Entity Store history). Other temporal signals (e.g. investigation
status, watchlist) are left as hooks for later.

- **Behavorial Analysis** : Uses security alerts tied to each entity
over a 7 day window. It highlights severity (critical/high vs medium vs
low), alert volume spikes, and cases where many distinct detection rules
fire on the same entity (multi-tactic style activity).

All three modules are registered with the lead generation engine and
contribute to the same pipeline: entity retrieval -> observation
collection -> scoring -> grouping and synthesis. No new API surface;
this builds on the existing generate/find lead routes and the foundation
from #255272.

Till the foundation PR gets merge, use the below comparison link to only
see the changes for observation modules


abhishekbhatia1710/kibana@ea-15946-lead-gen-foundation...ea-15947-observation-modules


Testing Steps : 

The `POST /internal/entity_analytics/leads/generate` and `GET
/internal/entity_analytics/leads`  are only available for now.

1. Entity Store V2 enabled with some user/host entities.
2. Risk Engine enabled and risk score data (for the risk analysis module
to find scores). (document generator)
3. Some detection rules firing alerts against those entities (for the
alert analysis module). (This i created using gemini)

Then call `POST /internal/entity_analytics/leads/generate`. Generated
leads show up via `GET /internal/entity_analytics/leads`.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…t to get data for lead generation (#256628)

## Summary

- Introduces a shared **Entity Retriever** that centralises Entity Store
V2 fetching, paginated via `search_after` for all users/hosts, or
targeted by name, replacing duplicated inline logic in the
generate-leads route
- Introduces an **Entity Enricher** that pre-fetches risk score history
(90-day daily averages), alert summaries (severity breakdown, top rules,
top alerts), asset criticality, and privileged status for a batch of
entities
- Refactors the generate-leads route to delegate entity fetching to the
retriever, removing ~70 lines of inline code
- Updates the barrel file to export both new services and their types


Closes : elastic/security-team#15948

This PR is 3rd part of the lead generation feature being built under the
[Entity Analytics Lead Generation
epic](elastic/security-team#14401). It builds
on the foundation and observation modules from #255272 and #256156.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…t to get data for lead generation (elastic#256628)

## Summary

- Introduces a shared **Entity Retriever** that centralises Entity Store
V2 fetching, paginated via `search_after` for all users/hosts, or
targeted by name, replacing duplicated inline logic in the
generate-leads route
- Introduces an **Entity Enricher** that pre-fetches risk score history
(90-day daily averages), alert summaries (severity breakdown, top rules,
top alerts), asset criticality, and privileged status for a batch of
entities
- Refactors the generate-leads route to delegate entity fetching to the
retriever, removing ~70 lines of inline code
- Updates the barrel file to export both new services and their types


Closes : elastic/security-team#15948

This PR is 3rd part of the lead generation feature being built under the
[Entity Analytics Lead Generation
epic](elastic/security-team#14401). It builds
on the foundation and observation modules from elastic#255272 and elastic#256156.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…t to get data for lead generation (elastic#256628)

## Summary

- Introduces a shared **Entity Retriever** that centralises Entity Store
V2 fetching, paginated via `search_after` for all users/hosts, or
targeted by name, replacing duplicated inline logic in the
generate-leads route
- Introduces an **Entity Enricher** that pre-fetches risk score history
(90-day daily averages), alert summaries (severity breakdown, top rules,
top alerts), asset criticality, and privileged status for a batch of
entities
- Refactors the generate-leads route to delegate entity fetching to the
retriever, removing ~70 lines of inline code
- Updates the barrel file to export both new services and their types


Closes : elastic/security-team#15948

This PR is 3rd part of the lead generation feature being built under the
[Entity Analytics Lead Generation
epic](elastic/security-team#14401). It builds
on the foundation and observation modules from elastic#255272 and elastic#256156.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

Resolve conflicts after PRs elastic#255272 (Foundation), elastic#256156 (Observation
Modules), elastic#256628 (Entity Retrieval), and elastic#257046 (CRUD API Routes)
were merged into main.

Key resolutions:
- Keep main's authoritative versions of observation modules with
  data-driven tier tables and RiskScoreDataClient integration
- Add scheduling-specific weight properties to module configs
- Wire RiskScoreDataClient through RunPipelineParams via dependency
  injection (route uses context, task creates from CoreStart)
- Preserve main's .keyword suffix fixes in lead_data_client ES queries
- Keep scheduling branch's single-file behavioral_analysis_module
  (consolidated from subdirectory structure) and remove stale subdir
- Delete generate_leads.test.ts (coverage moved to run_pipeline.test.ts)
- Preserve Task Manager registration, enable/disable route wiring,
  and run_pipeline shared orchestration from scheduling branch

Note: pre-commit hook bypassed because ESLint failures are from
upstream kbn-evals-suite-significant-events package (pre-existing
in main), not from lead generation changes.

…behavorial observation modules (elastic#256156)

## Summary

This PR adds three pluggable observation modules to the lead generation
pipeline. Each module collects signals about entities (users and hosts)
from different data sources. The engine combines these observations to
score and rank leads, and uses them when generating titles, tags, and
chat recommendations.

- **Risk Analysis** : Uses current risk scores on entity records and
(where available) risk score history. Flags entities that are high- or
critical-risk, and those with meaningful risk increases over 24 hours, 7
days, or 90 days. Privileged entities with elevated risk are called out
separately.

- **Temporal State Analysis** : Looks at how entity state changes over
time. Right now it focuses on **privilege escalation**: entities that
were not privileged in the past but are privileged in the latest view
(using Entity Store history). Other temporal signals (e.g. investigation
status, watchlist) are left as hooks for later.

- **Behavorial Analysis** : Uses security alerts tied to each entity
over a 7 day window. It highlights severity (critical/high vs medium vs
low), alert volume spikes, and cases where many distinct detection rules
fire on the same entity (multi-tactic style activity).

All three modules are registered with the lead generation engine and
contribute to the same pipeline: entity retrieval -> observation
collection -> scoring -> grouping and synthesis. No new API surface;
this builds on the existing generate/find lead routes and the foundation
from elastic#255272.

Till the foundation PR gets merge, use the below comparison link to only
see the changes for observation modules


abhishekbhatia1710/kibana@ea-15946-lead-gen-foundation...ea-15947-observation-modules


Testing Steps : 

The `POST /internal/entity_analytics/leads/generate` and `GET
/internal/entity_analytics/leads`  are only available for now.

1. Entity Store V2 enabled with some user/host entities.
2. Risk Engine enabled and risk score data (for the risk analysis module
to find scores). (document generator)
3. Some detection rules firing alerts against those entities (for the
alert analysis module). (This i created using gemini)

Then call `POST /internal/entity_analytics/leads/generate`. Generated
leads show up via `GET /internal/entity_analytics/leads`.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…t to get data for lead generation (elastic#256628)

## Summary

- Introduces a shared **Entity Retriever** that centralises Entity Store
V2 fetching, paginated via `search_after` for all users/hosts, or
targeted by name, replacing duplicated inline logic in the
generate-leads route
- Introduces an **Entity Enricher** that pre-fetches risk score history
(90-day daily averages), alert summaries (severity breakdown, top rules,
top alerts), asset criticality, and privileged status for a batch of
entities
- Refactors the generate-leads route to delegate entity fetching to the
retriever, removing ~70 lines of inline code
- Updates the barrel file to export both new services and their types


Closes : elastic/security-team#15948

This PR is 3rd part of the lead generation feature being built under the
[Entity Analytics Lead Generation
epic](elastic/security-team#14401). It builds
on the foundation and observation modules from elastic#255272 and elastic#256156.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…behavorial observation modules (elastic#256156)

## Summary

This PR adds three pluggable observation modules to the lead generation
pipeline. Each module collects signals about entities (users and hosts)
from different data sources. The engine combines these observations to
score and rank leads, and uses them when generating titles, tags, and
chat recommendations.

- **Risk Analysis** : Uses current risk scores on entity records and
(where available) risk score history. Flags entities that are high- or
critical-risk, and those with meaningful risk increases over 24 hours, 7
days, or 90 days. Privileged entities with elevated risk are called out
separately.

- **Temporal State Analysis** : Looks at how entity state changes over
time. Right now it focuses on **privilege escalation**: entities that
were not privileged in the past but are privileged in the latest view
(using Entity Store history). Other temporal signals (e.g. investigation
status, watchlist) are left as hooks for later.

- **Behavorial Analysis** : Uses security alerts tied to each entity
over a 7 day window. It highlights severity (critical/high vs medium vs
low), alert volume spikes, and cases where many distinct detection rules
fire on the same entity (multi-tactic style activity).

All three modules are registered with the lead generation engine and
contribute to the same pipeline: entity retrieval -> observation
collection -> scoring -> grouping and synthesis. No new API surface;
this builds on the existing generate/find lead routes and the foundation
from elastic#255272.

Till the foundation PR gets merge, use the below comparison link to only
see the changes for observation modules


abhishekbhatia1710/kibana@ea-15946-lead-gen-foundation...ea-15947-observation-modules


Testing Steps : 

The `POST /internal/entity_analytics/leads/generate` and `GET
/internal/entity_analytics/leads`  are only available for now.

1. Entity Store V2 enabled with some user/host entities.
2. Risk Engine enabled and risk score data (for the risk analysis module
to find scores). (document generator)
3. Some detection rules firing alerts against those entities (for the
alert analysis module). (This i created using gemini)

Then call `POST /internal/entity_analytics/leads/generate`. Generated
leads show up via `GET /internal/entity_analytics/leads`.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

…t to get data for lead generation (elastic#256628)

## Summary

- Introduces a shared **Entity Retriever** that centralises Entity Store
V2 fetching, paginated via `search_after` for all users/hosts, or
targeted by name, replacing duplicated inline logic in the
generate-leads route
- Introduces an **Entity Enricher** that pre-fetches risk score history
(90-day daily averages), alert summaries (severity breakdown, top rules,
top alerts), asset criticality, and privileged status for a batch of
entities
- Refactors the generate-leads route to delegate entity fetching to the
retriever, removing ~70 lines of inline code
- Updates the barrel file to export both new services and their types


Closes : elastic/security-team#15948

This PR is 3rd part of the lead generation feature being built under the
[Entity Analytics Lead Generation
epic](elastic/security-team#14401). It builds
on the foundation and observation modules from elastic#255272 and elastic#256156.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.