Bug description
Pilot crashes when attempting to pair a control plane cluster with a remote cluster after correcting a previously bad remote cluster configuration.
This sounded similar to #12376, which was marked as fixed by #12377, but this is still happening on Pilot versions built from latest master.
Expected behavior
Pilot should not panic in the case of applying a working configuration following the push of bad multicluster configuration.
Steps to reproduce the bug
Set up Istio in a multi-cluster configuration (single control plane cluster with a single remote cluster). We're essentially using the "flat network" setup, outlined in the docs.
Pair the control cluster to the remote cluster, but with a bad configuration. In our case, we had the following in the remote-cluster configuration:
apiVersion: v1
kind: Config
clusters:
- cluster:
server: ${SERVER}
certificate-authority-data: ${CA_DATA} # <--- This line is bad
insecure-skip-tls-verify: true
name: ${REMOTE_CLUSTER_NAME}
contexts:
- context:
cluster: ${REMOTE_CLUSTER_NAME}
user: ${REMOTE_CLUSTER_NAME}
name: ${REMOTE_CLUSTER_NAME}
current-context: ${REMOTE_CLUSTER_NAME}
preferences: {}
users:
- name: ${REMOTE_CLUSTER_NAME}
user:
token: ${TOKEN}Observe Pilot logging that it couldn't pair to the remote due to the bad configuration:
2019-04-18T00:50:14.537021Z info Processing add: istio-system/gke-foo-cluster
2019-04-18T00:50:14.537823Z info Adding new cluster member: gke-foo-cluster
2019-04-18T00:50:14.538005Z error error during create of kubernetes client interface for cluster: gke-foo-cluster specifying a root certificates file with the insecure flag is not allowed
Correct the bad configuration. In our case, we removed the CA configuration from the YAML file. Attempt to pair the clusters again. Pilot will panic.
2019-04-17T03:34:38.682567Z info Processing delete: istio-system/gke-foo-cluster
2019-04-17T03:34:38.682649Z info Deleting cluster member: gke-foo-cluster
2019-04-17T03:34:38.682665Z warn Registry is not found in the registries list, nothing to delete
2019-04-17T03:34:38.682839Z error Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/workspace/go/src/istio.io/istio/pilot/pkg/config/clusterregistry/multicluster.go:122
/workspace/go/src/istio.io/istio/pilot/pkg/config/clusterregistry/multicluster.go:74
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:265
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:212
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:187
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:174
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:156
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/workspace/go/src/istio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:156
/usr/local/go/src/runtime/asm_amd64.s:2361
Version (include the output of istioctl version --remote and kubectl version)
$ https_proxy=http://production-kube-master-proxy.cloudkitchens.internal:3128 kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:04:52Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-gke.10", GitCommit:"aaf0906400b5fc1d858ce0566a571e4f3ed06b9f", GitTreeState:"clean", BuildDate:"2019-03-30T19:30:48Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
$ istioctl version --remote
client version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.1"}
citadel version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
galley version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
ingressgateway version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"gcr.io/istio-release", BuildStatus:"Clean", GitTag:"1.1.3-1-gbf94d79"}
ingressgateway version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"gcr.io/istio-release", BuildStatus:"Clean", GitTag:"1.1.3-1-gbf94d79"}
ingressgateway version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"gcr.io/istio-release", BuildStatus:"Clean", GitTag:"1.1.3-1-gbf94d79"}
ingressgateway version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"gcr.io/istio-release", BuildStatus:"Clean", GitTag:"1.1.3-1-gbf94d79"}
ingressgateway version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"gcr.io/istio-release", BuildStatus:"Clean", GitTag:"1.1.3-1-gbf94d79"}
pilot version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
policy version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
sidecar-injector version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
telemetry version: version.BuildInfo{Version:"release-1.1-20190413-09-16", GitRevision:"bf94d79388b47959a2ce2d3212ba59f682dd2e11-dirty", User:"root", Host:"e5ba867a-5dcc-11e9-b523-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.3-1-gbf94d79"}
How was Istio installed?
Using the install directory from the git repo at version 1.1.2.
Environment where bug was observed (cloud vendor, OS, etc)
GKE. Container-Optimized OS VMs.
Bug description
Pilot crashes when attempting to pair a control plane cluster with a remote cluster after correcting a previously bad remote cluster configuration.
This sounded similar to #12376, which was marked as fixed by #12377, but this is still happening on Pilot versions built from latest master.
Expected behavior
Pilot should not panic in the case of applying a working configuration following the push of bad multicluster configuration.
Steps to reproduce the bug
Set up Istio in a multi-cluster configuration (single control plane cluster with a single remote cluster). We're essentially using the "flat network" setup, outlined in the docs.
Pair the control cluster to the remote cluster, but with a bad configuration. In our case, we had the following in the remote-cluster configuration:
Observe Pilot logging that it couldn't pair to the remote due to the bad configuration:
Correct the bad configuration. In our case, we removed the CA configuration from the YAML file. Attempt to pair the clusters again. Pilot will panic.
Version (include the output of
istioctl version --remoteandkubectl version)How was Istio installed?
Using the install directory from the git repo at version 1.1.2.
Environment where bug was observed (cloud vendor, OS, etc)
GKE. Container-Optimized OS VMs.