In #18758 the decision was made to change the config structure of the Elastic Agent. This issue is to track the changes needed across all parts of the Stack. It is split up in different phases so the changes can be done in small chunks and it can be tested everything still works as expected.

Phase 1 - Remove data source

In Phase 1, the datasources array is removed from the config and all the configs on the datasource level are moved to the input level.
Before:

datasources:
  - namespace: default
    use_output: default
    inputs:
      - type: system/metrics
        streams:
          - metricset: cpu
            dataset: system.cpu

After:

inputs:
 - namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset: system.cpu

Changes to be made:

• Update config processing in Elastic Agent [Ingest Manager] New structure of agent configuration #19128
• Update config generation in Ingest Manager (Kibana). Datasources on saved object level should not be removed, only what is generated for the Agent is changed. [Ingest Manager] Replace datasources with inputs when generating agent config kibana#69226

Phase 2 - Adjust dataset naming

Based on elastic/package-registry#482 configs related to datasets should be adjusted

Before

inputs:
 - namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset: system.cpu

After

inputs:
 - dataset.namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset.name: system.cpu

• Add support for renamed configs on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
• Update Kibana generation of configs accordingly [Ingest Manager] Replace datasources with inputs when generating agent config kibana#69226
• Adjust packages in package-storage
• Adjust package-registry and manifest
• Update integrations package generation if needed

Phase 3 - Remove datasources

The naming of datasources should be renamed in Kibana, the package-registry and the packages.

• Rename datasources (SO) in Kibana [Ingest Manager] Rename data sources to package configs kibana#70259
• Rename datasources list in package manifest to config_templatesUpdate package output structure package-registry#514
• Update packages in package-storage to new format
• Update integrations repositories to the new structure Update package output structure package-registry#514
• Update Kibana to support package registry changes [Ingest Manager] Support updated package output structure kibana#69864

Phase 4 - Add support for dataset.type

Currently the dataset.type is inherited from the input.type but this might not always correct. In the future it should be possible to set the dataset.type independently. Below is an example on how this could look (example does not make much sense):

inputs:
 - dataset.namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset.name: system.cpu
       dataset.type: logs

• Add support for it on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
• Adjust package-registry and manifest as needed Update package output structure package-registry#514
• Add dataset.type to Kibana config generation [Ingest Manager] Support updated package output structure kibana#69864