-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Hard coded metadata endpoint breaks logging under GKE managed Istio #7517
Copy link
Copy link
Closed
Labels
api: loggingIssues related to the Cloud Logging API.Issues related to the Cloud Logging API.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Description
Environment details
- API: GCE / GKE internal metadata endpoint
- OS: anything running in GKE with managed istio enabled.
- Python version: 3.7.2 (container, no venv)
- google-cloud-logging version: 1.9.0 (out of date but I double checked on master branch and
metadatahostname is present there too).
Steps to reproduce
(please bear with me, I do get to the point!)
- Launch a GKE cluster and tick the managed istio box to enable it.
- Add the label
istio-inejction: enabledto the default namespace to allow istio to control traffic on workloads deployed there. - Launch a container into the namespace which will now have the istio/envoy sidecar automatically added to it. (eg.
kubectl run -i --tty busybox --image=busybox --restart=Never -- sh) - Attempt to access the hostname
metadata.
wget http://metadata/computeMetadata/v1/instance/attributes/cluster-name --header "Metadata-Flavor: Google"
Connecting to metadata (169.254.169.254:80)
wget: server returned error: HTTP/1.1 404 Not Found
This is the result of Istio's default policy of blocking all outbound requests until they are whitelisted.
Since istio/istio#10694 will not accept non FQDN's the hardcoded metadata endpoint in the python logging client fails.
This manifests as logs showing up in stackdriver without the expected GCE / GKE annotations of instance name, cluster name, etc.
The approach from other google libraries (eg. googleapis/google-cloud-java#4269 ) is to update to the fully qualified metadata.google.internal or just hit the magic IP address directly which Istio has no problems whitelisting.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
api: loggingIssues related to the Cloud Logging API.Issues related to the Cloud Logging API.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.