[2019-04-30 20:53:46.480][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-04-30T20:53:47.352312Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:49.352546Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:51.352591Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:53.352597Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:55.352384Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:57.352482Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:59.352326Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:01.352462Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:03.352228Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-04-30 20:54:04.102][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-04-30T20:54:05.351974Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:07.352561Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:09.352055Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:11.352231Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:13.352079Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:15.351966Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-04-30 20:54:15.561][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.561][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.564][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.568][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.568][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x10
[2019-04-30 20:54:15.568][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:67] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-04-30 20:54:15.569][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fc0f3df8390]
[2019-04-30 20:54:15.573][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Server::ListenerImpl::initialize() [0x872ca2]
[2019-04-30 20:54:15.576][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0x8755ef]
[2019-04-30 20:54:15.579][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x893038]
[2019-04-30 20:54:15.582][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Config::GrpcMuxSubscriptionImpl<>::onConfigUpdate() [0x89677d]
[2019-04-30 20:54:15.585][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Config::GrpcMuxImpl::handleResponse() [0xa05a16]
[2019-04-30 20:54:15.588][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0xa0663e]
[2019-04-30 20:54:15.591][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Grpc::AsyncStreamImpl::onData() [0xa2df91]
[2019-04-30 20:54:15.594][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::AsyncStreamImpl::encodeData() [0xa31b64]
[2019-04-30 20:54:15.597][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0xa8ab11]
[2019-04-30 20:54:15.600][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_8::__invoke() [0xa8eb5c]
[2019-04-30 20:54:15.602][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: nghttp2_session_on_data_received [0xa9720f]
[2019-04-30 20:54:15.605][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: nghttp2_session_mem_recv [0xa99037]
[2019-04-30 20:54:15.608][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0xa8a476]
[2019-04-30 20:54:15.611][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::CodecClient::onData() [0xa24e46]
[2019-04-30 20:54:15.614][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::CodecClient::CodecReadFilter::onData() [0xa258ed]
[2019-04-30 20:54:15.617][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Network::FilterManagerImpl::onRead() [0x8b920a]
[2019-04-30 20:54:15.620][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Network::ConnectionImpl::onReadReady() [0x8b5e1e]
[2019-04-30 20:54:15.623][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Network::ConnectionImpl::onFileEvent() [0x8b58f2]
[2019-04-30 20:54:15.626][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x8b03d5]
[2019-04-30 20:54:15.629][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: event_process_active_single_queue [0xc52724]
[2019-04-30 20:54:15.632][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: event_base_loop [0xc50fac]
[2019-04-30 20:54:15.634][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Event::DispatcherImpl::run() [0x8af793]
[2019-04-30 20:54:15.637][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: Envoy::Server::InstanceImpl::run() [0x867bde]
[2019-04-30 20:54:15.640][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: Envoy::MainCommonBase::run() [0x551889]
[2019-04-30 20:54:15.643][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: main [0x550952]
[2019-04-30 20:54:15.643][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #26: [0x7fc0f3734830]
2019-04-30T20:54:15.706886Z warn Epoch 0 terminated with an error: signal: segmentation fault (core dumped)
2019-04-30T20:54:15.706924Z warn Aborted all epochs
2019-04-30T20:54:15.707045Z info Epoch 0: set retry delay to 200ms, budget to 9
2019-04-30T20:54:15.907253Z info Reconciling retry (budget 9)
2019-04-30T20:54:15.907334Z info Epoch 0 starting
Still investigating root cause and how to consistently reproduce. The failures to connect to pilot are NOT the issue, I had purposely scaled pilot to 0, then the crash comes once connected to Pilot.
Steps to reproduce:
helm upgrade istio install/kubernetes/helm/istio --namespace istio-system --set global.proxy.image=istio/proxyv2:1.1.2Seems to only happen on ingressgateway
Still investigating root cause and how to consistently reproduce. The failures to connect to pilot are NOT the issue, I had purposely scaled pilot to 0, then the crash comes once connected to Pilot.