fatal error: concurrent map read and map write
goroutine 1187 [running]:
runtime.throw(0x1c43e3f, 0x21)
/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc42b1d4df8 sp=0xc42b1d4dd8 pc=0x42ac81
runtime.mapaccess2_faststr(0x1990e00, 0xc4258bae10, 0xc4233af810, 0x2e, 0xc4233af810, 0x2e)
/usr/local/go/src/runtime/hashmap_fast.go:270 +0x461 fp=0xc42b1d4e68 sp=0xc42b1d4df8 pc=0x40abb1
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).loadAssignmentsForClusterIsolated(0xc4205d86e0, 0xc423cc9500, 0xc42bb3cc00, 0xc4233af800, 0x3e, 0x0, 0xc4233af810)
/workspace/go/src/istio.io/istio/pilot/pkg/proxy/envoy/v2/eds.go:702 +0x109 fp=0xc42b1d51e8 sp=0xc42b1d4e68 pc=0x171a6e9
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushEds(0xc4205d86e0, 0xc42bb3cc00, 0xc423ae5880, 0x0, 0x16, 0x0)
/workspace/go/src/istio.io/istio/pilot/pkg/proxy/envoy/v2/eds.go:805 +0x72b fp=0xc42b1d5458 sp=0xc42b1d51e8 pc=0x171b9bb
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushConnection(0xc4205d86e0, 0xc423ae5880, 0xc426b2da40, 0x0, 0x0)
/workspace/go/src/istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:742 +0x5c5 fp=0xc42b1d5518 sp=0xc42b1d5458 pc=0x170bfa5
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).StreamAggregatedResources(0xc4205d86e0, 0x1de0700, 0xc423e13e90, 0x0, 0x0)
Hey,
On the latest nightly (1.1-20190423-09-16), one instance of istio-pilot died a horrible, fire-filled death.
It started with:
I've attached the whole pilot log for you.
bad.log