)

* Allow kibana_system role to read and manage logs streams

* remove logs index

* [CI] Auto commit changes from spotless

* add logs index

---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

…cations

* upstream/main: (35 commits)
  Create ARM bulk sqrI8 implementation (elastic#142461)
  Rework get-snapshots predicates (elastic#143161)
  Refactor downsampling fetchers and producers (elastic#140357)
  ESQL: Unmute test and add extra logging to generative test validation (elastic#143168)
  Fix metadata fields being nullified/loaded by unmapped_fields setting (elastic#143155)
  Determine remote cluster version (elastic#142494)
  Populate failure message for aborted clones (elastic#143206)
  Allow kibana_system role to read and manage logs streams (elastic#143053)
  Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsLength} elastic#143224
  Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsByteLength} elastic#143223
  Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:docs.DocsBitLength} elastic#143222
  Fix FloatVectorScorerSupplier bulkScore bug (elastic#143211)
  ESQL: Add data node execution for external sources (elastic#143209)
  [ESQL] Cleanup commands docs (elastic#143058)
  [ML]Fix latest transforms disregarding updates when sort and sync fields are non-monotonic (elastic#142856)
  Mute org.elasticsearch.index.mapper.IpFieldMapperTests testSyntheticSourceInObject elastic#143212
  Tests: Fix StoreDirectoryMetricsIT (elastic#143084)
  ESQL: Add distribution strategy for external sources (elastic#143194)
  CSV IT spec (elastic#142585)
  Fix VectorScorerOSQBenchmark.score to read corrections properly (elastic#143137)
  ...

)

* Allow kibana_system role to read and manage logs streams

* remove logs index

* [CI] Auto commit changes from spotless

* add logs index

---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

## Summary

Related to elastic/streams-program#402

This PR allows configuring wired streams definitions in Kibana config
and applying them on startup.

`StreamsClient` has been refactored to be able to work without Kibana
requests, making it possible to pass both scoped and unscoped dependency
clients.

This change depends on
elastic/elasticsearch#143053

**Sample stream definition:**
```
xpack.streams:
  preconfigured:
    enabled: true
    stream_definitions:
    - name: logs.ecs
      dashboards: []
      queries: []
      rules: []
      stream:
        description: ""
        query_streams: []
        ingest:
          lifecycle:
            dsl: {}
          processing:
            steps: []
          settings: {}
          failure_store:
            lifecycle:
              enabled:
                data_retention: '20d'
          wired:
            routing:
            - destination: logs.ecs.child1
              where:
                field: 'resource.attributes.host.name'
                startsWith: 'filebeat'
              status: 'enabled'
``` 
> [!NOTE]
> Just setting `xpack.streams.preconfigured.enabled: true` will only
create the root `logs.ecs` and `logs.otel` streams.

> [!IMPORTANT]
> Since the RulesClient cannot be made unscoped, `queries` and `rules`
should always be empty. This is a limitation of the config-based streams
for now.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for preconfigured streams, allowing streams to be
defined and automatically enabled at startup with customizable field
sets (ECS or base fields).

* **Configuration**
* Extended configuration schema to include preconfiguration options for
managing stream definitions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>

## Summary

Related to elastic/streams-program#402

This PR allows configuring wired streams definitions in Kibana config
and applying them on startup.

`StreamsClient` has been refactored to be able to work without Kibana
requests, making it possible to pass both scoped and unscoped dependency
clients.

This change depends on
elastic/elasticsearch#143053

**Sample stream definition:**
```
xpack.streams:
  preconfigured:
    enabled: true
    stream_definitions:
    - name: logs.ecs
      dashboards: []
      queries: []
      rules: []
      stream:
        description: ""
        query_streams: []
        ingest:
          lifecycle:
            dsl: {}
          processing:
            steps: []
          settings: {}
          failure_store:
            lifecycle:
              enabled:
                data_retention: '20d'
          wired:
            routing:
            - destination: logs.ecs.child1
              where:
                field: 'resource.attributes.host.name'
                startsWith: 'filebeat'
              status: 'enabled'
``` 
> [!NOTE]
> Just setting `xpack.streams.preconfigured.enabled: true` will only
create the root `logs.ecs` and `logs.otel` streams.

> [!IMPORTANT]
> Since the RulesClient cannot be made unscoped, `queries` and `rules`
should always be empty. This is a limitation of the config-based streams
for now.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for preconfigured streams, allowing streams to be
defined and automatically enabled at startup with customizable field
sets (ECS or base fields).

* **Configuration**
* Extended configuration schema to include preconfiguration options for
managing stream definitions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>

## Summary

Related to https://github.com/elastic/streams-program/issues/402

This PR allows configuring wired streams definitions in Kibana config
and applying them on startup.

`StreamsClient` has been refactored to be able to work without Kibana
requests, making it possible to pass both scoped and unscoped dependency
clients.

This change depends on
elastic/elasticsearch#143053

**Sample stream definition:**
```
xpack.streams:
  preconfigured:
    enabled: true
    stream_definitions:
    - name: logs.ecs
      dashboards: []
      queries: []
      rules: []
      stream:
        description: ""
        query_streams: []
        ingest:
          lifecycle:
            dsl: {}
          processing:
            steps: []
          settings: {}
          failure_store:
            lifecycle:
              enabled:
                data_retention: '20d'
          wired:
            routing:
            - destination: logs.ecs.child1
              where:
                field: 'resource.attributes.host.name'
                startsWith: 'filebeat'
              status: 'enabled'
``` 
> [!NOTE]
> Just setting `xpack.streams.preconfigured.enabled: true` will only
create the root `logs.ecs` and `logs.otel` streams.

> [!IMPORTANT]
> Since the RulesClient cannot be made unscoped, `queries` and `rules`
should always be empty. This is a limitation of the config-based streams
for now.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for preconfigured streams, allowing streams to be
defined and automatically enabled at startup with customizable field
sets (ECS or base fields).

* **Configuration**
* Extended configuration schema to include preconfiguration options for
managing stream definitions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>