-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Log entry of resource type "k8s_container": "location" label set incorrectly #9083
Description
When using the Go library to create container log entry (resource type k8s_container), the logging location label is set to the Node zone instead of the cluster location.
google-cloud-go/logging/resource.go
Line 187 in 477ccee
| "location": zone, |
Cloud Monitoring documentation covers this:
https://cloud.google.com/monitoring/api/resources#tag_k8s_container
location: The physical location of the cluster that contains the container.
For a GKE regional cluster location should be set in the cluster region and the same logic applies for the zonal cluster.
Note this is set correctly for logs going through the GKE logging pipeline as well as for some of other client library such as ruby and C++:
- https://github.com/googleapis/google-api-ruby-client/blob/3c1c3311a9179bbb3e3b17f1939d180748e5e6d0/generated/google-apis-monitoring_v3/lib/google/apis/monitoring_v3/classes.rb#L1694
- https://github.com/googleapis/google-cloud-cpp/blob/850564f330ac6e918d6d40ac1ab17a200282a196/google/cloud/opentelemetry/internal/monitored_resource.cc#L88
I believe the reason why the location label is set to the Cluster location in the first place is that this along with the cluster name is required to uniquely identify the cluster. GKE allows the cluster to have the same name if the location is different, so the name alone is not enough.
The location label being set incorrectly can result in the container log not showing up as expected in Cloud console due to them not matching the expected log filter. In some cases logs could potentially be attributed to the wrong cluster.
Expected behavior
Log entry created using k8s_container resource type have the location label set to the cluster location
Actual behavior
Log entry created using k8s_container resource type and the location label set to the Node zone