-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Envoy Crash in EDS update #6282
Copy link
Copy link
Closed
Labels
bugstalestalebot believes this issue/PR has not been touched recentlystalebot believes this issue/PR has not been touched recently
Description
Title: Envoy crash during EDS update
Description:
We saw an Envoy crash (log output below). It seems the crash is triggered by an EDS update when updating host priorities.
envoy version: 32bdce3
log:
2019-03-13_19:14:47.55623 [2019-03-13 19:14:47.556][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x8
2019-03-13_19:14:47.55629 [2019-03-13 19:14:47.556][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
2019-03-13_19:14:47.55669 [2019-03-13 19:14:47.556][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fac05eec5d0]
2019-03-13_19:14:47.63158 [2019-03-13 19:14:47.631][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Upstream::PrioritySetImpl::updateHosts() [0x7fc927]
2019-03-13_19:14:47.63473 [2019-03-13 19:14:47.634][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Upstream::PriorityStateManager::updateClusterPrioritySet() [0x80716d]
2019-03-13_19:14:47.63743 [2019-03-13 19:14:47.637][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Upstream::EdsClusterImpl::updateHostsPerLocality() [0x812372]
2019-03-13_19:14:47.64011 [2019-03-13 19:14:47.640][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Upstream::EdsClusterImpl::onConfigUpdate() [0x812e70]
2019-03-13_19:14:47.64277 [2019-03-13 19:14:47.642][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Config::GrpcMuxSubscriptionImpl<>::onConfigUpdate() [0x814411]
2019-03-13_19:14:47.64555 [2019-03-13 19:14:47.645][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Config::GrpcMuxImpl::handleResponse() [0x819b10]
2019-03-13_19:14:47.64819 [2019-03-13 19:14:47.648][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0x81a1da]
2019-03-13_19:14:47.65084 [2019-03-13 19:14:47.650][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Grpc::AsyncStreamImpl::onData() [0x83c8a6]
2019-03-13_19:14:47.65346 [2019-03-13 19:14:47.653][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::AsyncStreamImpl::encodeData() [0x843695]
2019-03-13_19:14:47.65610 [2019-03-13 19:14:47.656][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x89a7b1]
2019-03-13_19:14:47.65887 [2019-03-13 19:14:47.658][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: nghttp2_session_on_data_received [0x89fea0]
2019-03-13_19:14:47.66143 [2019-03-13 19:14:47.661][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: nghttp2_session_mem_recv [0x8a49db]
2019-03-13_19:14:47.66409 [2019-03-13 19:14:47.664][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x89ba17]
2019-03-13_19:14:47.66655 [2019-03-13 19:14:47.666][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::CodecClient::onData() [0x833591]
2019-03-13_19:14:47.66894 [2019-03-13 19:14:47.668][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x83376d]
2019-03-13_19:14:47.67143 [2019-03-13 19:14:47.671][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Network::FilterManagerImpl::onContinueReading() [0x6e369c]
2019-03-13_19:14:47.67428 [2019-03-13 19:14:47.674][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Network::ConnectionImpl::onReadReady() [0x6e005a]
2019-03-13_19:14:47.67694 [2019-03-13 19:14:47.676][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Network::ConnectionImpl::onFileEvent() [0x6e076a]
2019-03-13_19:14:47.67944 [2019-03-13 19:14:47.679][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN() [0x6da05a]
2019-03-13_19:14:47.68178 [2019-03-13 19:14:47.681][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: event_process_active_single_queue.isra.29 [0xa1785f]
2019-03-13_19:14:47.68409 [2019-03-13 19:14:47.684][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: event_base_loop [0xa17e8f]
2019-03-13_19:14:47.68604 [2019-03-13 19:14:47.686][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Event::DispatcherImpl::run() [0x6d8fc3]
2019-03-13_19:14:47.68793 [2019-03-13 19:14:47.687][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: Envoy::Server::InstanceImpl::run() [0x695056]
2019-03-13_19:14:47.69004 [2019-03-13 19:14:47.690][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: Envoy::MainCommonBase::run() [0x45a811]
2019-03-13_19:14:47.69200 [2019-03-13 19:14:47.691][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: main [0x418f43]
2019-03-13_19:14:47.69230 [2019-03-13 19:14:47.692][175231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #26: __libc_start_main [0x7fac058303d5]
I can follow up with more information later
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugstalestalebot believes this issue/PR has not been touched recentlystalebot believes this issue/PR has not been touched recently