Skip to content

Hard coded metadata endpoint breaks logging under GKE managed Istio #7517

@rsalmond

Description

@rsalmond

Environment details

  1. API: GCE / GKE internal metadata endpoint
  2. OS: anything running in GKE with managed istio enabled.
  3. Python version: 3.7.2 (container, no venv)
  4. google-cloud-logging version: 1.9.0 (out of date but I double checked on master branch and metadata hostname is present there too).

Steps to reproduce

(please bear with me, I do get to the point!)

  1. Launch a GKE cluster and tick the managed istio box to enable it.
  2. Add the label istio-inejction: enabled to the default namespace to allow istio to control traffic on workloads deployed there.
  3. Launch a container into the namespace which will now have the istio/envoy sidecar automatically added to it. (eg. kubectl run -i --tty busybox --image=busybox --restart=Never -- sh)
  4. Attempt to access the hostname metadata.
wget http://metadata/computeMetadata/v1/instance/attributes/cluster-name --header "Metadata-Flavor: Google"
Connecting to metadata (169.254.169.254:80)
wget: server returned error: HTTP/1.1 404 Not Found

This is the result of Istio's default policy of blocking all outbound requests until they are whitelisted.

Since istio/istio#10694 will not accept non FQDN's the hardcoded metadata endpoint in the python logging client fails.

This manifests as logs showing up in stackdriver without the expected GCE / GKE annotations of instance name, cluster name, etc.

The approach from other google libraries (eg. googleapis/google-cloud-java#4269 ) is to update to the fully qualified metadata.google.internal or just hit the magic IP address directly which Istio has no problems whitelisting.

Metadata

Metadata

Assignees

Labels

api: loggingIssues related to the Cloud Logging API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions