Skip to content

Segfault on hotrestart: in sendDiscoveryRequest() during InstanceImpl::terminate() #8923

@mandarjog

Description

@mandarjog

This using latest nearly the latest version of Envoy built with Istio-proxy.

sendDiscoveryRequest is being sent during the Envoy::Server::InstanceImpl::terminate() sequence.

@htuch @lambdai

2019-11-07T01:06:37.544948Z	info	Envoy proxy is ready
2019-11-07T01:07:11.889669Z	info	watchFileEvents: "/etc/certs/..2019_11_07_01_07_11.996505983": CREATE
2019-11-07T01:07:11.889771Z	info	watchFileEvents: "/etc/certs/..2019_11_07_01_07_11.996505983": MODIFY|ATTRIB
2019-11-07T01:07:11.889779Z	info	watchFileEvents: "/etc/certs/cert-chain.pem": CREATE
2019-11-07T01:07:11.889830Z	info	watchFileEvents: "/etc/certs/key.pem": CREATE
2019-11-07T01:07:11.889857Z	info	watchFileEvents: "/etc/certs/root-cert.pem": CREATE
2019-11-07T01:07:11.889870Z	info	watchFileEvents: "/etc/certs/..data_tmp": RENAME
2019-11-07T01:07:11.889899Z	info	watchFileEvents: "/etc/certs/..data": CREATE
2019-11-07T01:07:11.889917Z	info	watchFileEvents: "/etc/certs/..2019_11_07_01_05_40.526689452": DELETE
2019-11-07T01:07:21.889871Z	info	watchFileEvents: notifying
2019-11-07T01:07:21.890044Z	info	Received new config, creating new Envoy epoch 1
2019-11-07T01:07:21.890061Z	info	waiting for epoch 0 to go live before performing a hot restart
2019-11-07T01:07:21.890971Z	info	Epoch 1 starting
2019-11-07T01:07:21.900275Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.56.8.15~istio-ingressgateway-647456cc7-9g8hz.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 1)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
[Envoy (Epoch 0)] [2019-11-07 01:07:21.921][20][warning][main] [external/envoy/source/server/server.cc:633] shutting down admin due to child startup
[Envoy (Epoch 0)] [2019-11-07 01:07:21.921][20][warning][main] [external/envoy/source/server/server.cc:639] terminating parent process
[Envoy (Epoch 1)] [2019-11-07 01:07:21.950][156][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 1)] [2019-11-07 01:07:21.950][156][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[Envoy (Epoch 0)] [2019-11-07 01:08:23.256][20][warning][main] [external/envoy/source/server/server.cc:514] caught SIGTERM
[Envoy (Epoch 0)] [2019-11-07 01:08:23.290][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2019-11-07 01:08:23.296][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x55c9612bd040
[Envoy (Epoch 0)] [2019-11-07 01:08:23.296][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[Envoy (Epoch 0)] [2019-11-07 01:08:23.296][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 3bc62393c89fcb33889fca77c244db1fc3cef05b/1.12.0/Clean/RELEASE/BoringSSL
[Envoy (Epoch 0)] [2019-11-07 01:08:23.297][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f0ef0bdd890]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.304][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Config::GrpcMuxImpl::sendDiscoveryRequest() [0x55c95db3a6c4]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.311][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::Config::GrpcMuxImpl::GrpcMuxWatchImpl::~GrpcMuxWatchImpl() [0x55c95db3ca6c]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.318][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::Config::GrpcMuxImpl::GrpcMuxWatchImpl::~GrpcMuxWatchImpl() [0x55c95db3cb0e]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.325][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Config::GrpcSubscriptionImpl::~GrpcSubscriptionImpl() [0x55c95db28054]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.331][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: Envoy::Secret::SdsApi::~SdsApi() [0x55c95d9c5be1]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.338][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Extensions::TransportSockets::Tls::ContextConfigImpl::~ContextConfigImpl() [0x55c95d9259bd]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.345][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: Envoy::Extensions::TransportSockets::Tls::ClientContextConfigImpl::~ClientContextConfigImpl() [0x55c95d9273b2]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.351][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: Envoy::Extensions::TransportSockets::Tls::ClientSslSocketFactory::~ClientSslSocketFactory() [0x55c95d922af7]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.358][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: Envoy::Upstream::TransportSocketMatcherImpl::~TransportSocketMatcherImpl() [0x55c95db4312a]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.365][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() [0x55c95db0f6f6]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.371][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() [0x55c95db0f74e]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.378][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: Envoy::Upstream::HostDescriptionImpl::~HostDescriptionImpl() [0x55c95cb3507d]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.385][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #13: Envoy::Upstream::HostImpl::~HostImpl() [0x55c95cb3c89d]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.391][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #14: std::__1::__shared_ptr_pointer<>::__on_zero_shared() [0x55c95cb35a3e]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.398][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #15: Envoy::Upstream::HostSetImpl::~HostSetImpl() [0x55c95ce0a8a6]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.405][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #16: Envoy::Upstream::HostSetImpl::~HostSetImpl() [0x55c95db0f2de]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.411][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #17: Envoy::Upstream::PrioritySetImpl::~PrioritySetImpl() [0x55c95cb339a0]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.418][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #18: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::~ClusterEntry() [0x55c95d9e996e]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.425][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #19: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::~ThreadLocalClusterManagerImpl() [0x55c95d9e6e9d]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.432][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #20: Envoy::ThreadLocal::InstanceImpl::shutdownThread() [0x55c95d9563e4]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.438][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #21: Envoy::Server::InstanceImpl::terminate() [0x55c95d95fa55]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.445][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #22: Envoy::Server::InstanceImpl::run() [0x55c95d966adb]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.452][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #23: Envoy::MainCommonBase::run() [0x55c95caf4338]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.458][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #24: main [0x55c95caf3312]
[Envoy (Epoch 0)] [2019-11-07 01:08:23.458][20][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #25: __libc_start_main [0x7f0ef07fbb97]
2019-11-07T01:08:23.713395Z	error	Epoch 0 exited with error: signal: segmentation fault (core dumped)

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions