Skip to content

Bad listener config segfaults istio-proxy 1.1.2 #13735

@howardjohn

Description

@howardjohn

Steps to reproduce:

  • Install Istio 1.1.4
  • helm upgrade istio install/kubernetes/helm/istio --namespace istio-system --set global.proxy.image=istio/proxyv2:1.1.2
  • Some variation of killing pilot and ingressgateway

Seems to only happen on ingressgateway

[2019-04-30 20:53:46.480][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-04-30T20:53:47.352312Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:49.352546Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:51.352591Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:53.352597Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:55.352384Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:57.352482Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:53:59.352326Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:01.352462Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:03.352228Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-04-30 20:54:04.102][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-04-30T20:54:05.351974Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:07.352561Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:09.352055Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:11.352231Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:13.352079Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-30T20:54:15.351966Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-04-30 20:54:15.561][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.561][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.564][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.568][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-30 20:54:15.568][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x10
[2019-04-30 20:54:15.568][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:67] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-04-30 20:54:15.569][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fc0f3df8390]
[2019-04-30 20:54:15.573][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Server::ListenerImpl::initialize() [0x872ca2]
[2019-04-30 20:54:15.576][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0x8755ef]
[2019-04-30 20:54:15.579][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x893038]
[2019-04-30 20:54:15.582][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Config::GrpcMuxSubscriptionImpl<>::onConfigUpdate() [0x89677d]
[2019-04-30 20:54:15.585][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Config::GrpcMuxImpl::handleResponse() [0xa05a16]
[2019-04-30 20:54:15.588][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0xa0663e]
[2019-04-30 20:54:15.591][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Grpc::AsyncStreamImpl::onData() [0xa2df91]
[2019-04-30 20:54:15.594][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::AsyncStreamImpl::encodeData() [0xa31b64]
[2019-04-30 20:54:15.597][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0xa8ab11]
[2019-04-30 20:54:15.600][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_8::__invoke() [0xa8eb5c]
[2019-04-30 20:54:15.602][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: nghttp2_session_on_data_received [0xa9720f]
[2019-04-30 20:54:15.605][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: nghttp2_session_mem_recv [0xa99037]
[2019-04-30 20:54:15.608][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0xa8a476]
[2019-04-30 20:54:15.611][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::CodecClient::onData() [0xa24e46]
[2019-04-30 20:54:15.614][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::CodecClient::CodecReadFilter::onData() [0xa258ed]
[2019-04-30 20:54:15.617][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Network::FilterManagerImpl::onRead() [0x8b920a]
[2019-04-30 20:54:15.620][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Network::ConnectionImpl::onReadReady() [0x8b5e1e]
[2019-04-30 20:54:15.623][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Network::ConnectionImpl::onFileEvent() [0x8b58f2]
[2019-04-30 20:54:15.626][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x8b03d5]
[2019-04-30 20:54:15.629][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: event_process_active_single_queue [0xc52724]
[2019-04-30 20:54:15.632][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: event_base_loop [0xc50fac]
[2019-04-30 20:54:15.634][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Event::DispatcherImpl::run() [0x8af793]
[2019-04-30 20:54:15.637][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: Envoy::Server::InstanceImpl::run() [0x867bde]
[2019-04-30 20:54:15.640][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: Envoy::MainCommonBase::run() [0x551889]
[2019-04-30 20:54:15.643][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: main [0x550952]
[2019-04-30 20:54:15.643][18][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #26: [0x7fc0f3734830]
2019-04-30T20:54:15.706886Z     warn    Epoch 0 terminated with an error: signal: segmentation fault (core dumped)
2019-04-30T20:54:15.706924Z     warn    Aborted all epochs
2019-04-30T20:54:15.707045Z     info    Epoch 0: set retry delay to 200ms, budget to 9
2019-04-30T20:54:15.907253Z     info    Reconciling retry (budget 9)
2019-04-30T20:54:15.907334Z     info    Epoch 0 starting

Still investigating root cause and how to consistently reproduce. The failures to connect to pilot are NOT the issue, I had purposely scaled pilot to 0, then the crash comes once connected to Pilot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions