This issue is a follow up of the discussions in elastic/integrations#4236 and #199

We need a source of truth for the mappings of the fields added by data providers, mainly for Elastic Agent and the processors it includes. These mappings will be used by any feature that gets these fields populated. It could be mappings in package definitions, or Fleet when installing index templates for specific policies.

The proposal would be to create a new repository, called elastic-agent-common-schema, that includes the mappings in paths like <collector>/<feature>/fields.yml. So for example the mappings for the add_host_metadata processor of beats can be defined in beats/add_host_metadata/fields.yml.

Each one of these fields definitions would use the format of ECS flat fields (such as these files), and could be generated from other smaller files if needed, using ECS tooling. Alternatively, the format for fields in the package spec could be used, that is similar.

Its versioning should be aligned with Elastic Agent versioning.

Use cases

• Allow elastic-package to import definitions in packages in build time, similar to the external fields, but for groups of packages.
• Allow Fleet to add mappings for fields of custom processors.