Describe the feature:
Integrations should be able to produce dynamic data_stream.namespace values, rather than having this statically defined per policy.
Describe a specific use case for the feature:
Users sometimes would like to create separate data streams for their data -- e.g. split by APM service, service group, or service environment (dev/test/prod).
Splitting data streams like this enables users to apply different ILM or security policies depending on the application or group of applications. For example, one might wish to keep production logs for years for auditing or regulatory compliance; but logs for dev/test environments may be deleted in the order of days or weeks.
In APM, we would introduce configuration that would allow users to template the namespace. This would be a more restrictive form of the old output.elasticsearch.indices configuration in libbeat, where one could for example include %{[service.name]} to route events to service-specific indices.
Describe the feature:
Integrations should be able to produce dynamic
data_stream.namespacevalues, rather than having this statically defined per policy.Describe a specific use case for the feature:
Users sometimes would like to create separate data streams for their data -- e.g. split by APM service, service group, or service environment (dev/test/prod).
Splitting data streams like this enables users to apply different ILM or security policies depending on the application or group of applications. For example, one might wish to keep production logs for years for auditing or regulatory compliance; but logs for dev/test environments may be deleted in the order of days or weeks.
In APM, we would introduce configuration that would allow users to template the namespace. This would be a more restrictive form of the old
output.elasticsearch.indicesconfiguration in libbeat, where one could for example include%{[service.name]}to route events to service-specific indices.