https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

Several existing integrations using orchestrator.* fields, like the Kubernetes integration package, have established other conventions for capturing labels and annotations. The changes proposed here seem reasonable, but we should pull some other teams into the discussion to make sure there's alignment around the approach.

@elastic/obs-cloud-monitoring @elastic/obs-cloudnative-monitoring

Reading through the k8s docs, these key/values are guaranteed to be strings. Does using flattened make sense over keyword as the field types?

Instead of:

orchestrator.resource.annotation:"key1:value1"

Flattened fields parse out the leaf fields and users would instead query:

orchestrator.resource.annotation.key1:"value1"

Reading through the k8s docs, these key/values are guaranteed to be strings. Does using flattened make sense over keyword as the field types?

Instead of:

orchestrator.resource.annotation:"key1:value1"

Flattened fields parse out the leaf fields and users would instead query:

orchestrator.resource.annotation.key1:"value1"

It seems wildcard searches are not possible with flattened? e.g. would "orchestrator.resource.annotation.key1:"partial*" be possible?

Seems the recent process.env_vars addition takes the same keyword approach. https://www.elastic.co/guide/en/ecs/current/ecs-process.html#field-process-env-vars

I see that the kubernetes integration has fields for annotations and labels, but all those fields seem to be custom and outside of ECS. https://github.com/elastic/integrations/blob/main/packages/kubernetes/data_stream/container/fields/base-fields.yml#L68

We are going to test using a "flattened" type for these new fields, and will report back here once we've determined if it's a viable way forward. Thanks!

I see that the kubernetes integration has fields for annotations and labels, but all those fields seem to be custom and outside of ECS

Indeed those fields are "custom", outside the ECS fields and mapped dynamically as they are discovered. I think to be able to search inside eg. "orchestrator.resource.label": [somelabel*] and be able to find an existing label it is really valuable use case for Kubernetes personas that access resources based on labels.

I see that the kubernetes integration has fields for annotations and labels, but all those fields seem to be custom and outside of ECS

Indeed those fields are "custom", outside the ECS fields and mapped dynamically as they are discovered. I think to be able to search inside eg. "orchestrator.resource.label": [somelabel*] and be able to find an existing label it is really valuable use case for Kubernetes personas that access resources based on labels.

Yea, if we go the "flattened" route, I guess we lose ability to do wildcard searches on label/annotation "keys". If kibana search bars autocomplete the key values, perhaps that is a non issue, though maybe is less flexible for protection rules etc..

Hello,

It seems wildcard searches are not possible with flattened? e.g. would "orchestrator.resource.annotation.key1:"partial*" be possible?

I just want to point out that the docs mention that you can't do wildcard search on field keys not field values. Like in the following extract from here

When querying, it is not possible to refer to field keys using wildcards, as in { "term": {"labels.time*": 1541457010}}

After testing the "flattened" type. It seems to have too many limitations for our use cases.
a) EQL won't allow it's use in rules
b) the keys of the object aren't autocompleted in kibana searches.
c) aggregations don't seem to be supported.

I think the PR as it stands (array of keywords e.g ["key:value", "key2:value2"] is the most flexible for our use case. cc @ebeahan

I think the PR as it stands (array of keywords e.g ["key:value", "key2:value2"] is the most flexible for our use case. cc

Sorry for the delay. Understood that flattened doesn't sound like the right fit here due to the limitations.

No objections to the changes from the ECS perspective, and @gizas supported the approach here.

@kgeller we're past the 8.8 FF, but I suspect there's low risk to still including these changes into ECS 8.8. WDYT?

🥳

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Questions ?

Please refer to the Backport tool documentation