This repository was archived by the owner on Dec 16, 2020. It is now read-only.
This repository was archived by the owner on Dec 16, 2020. It is now read-only.
Crash under heavy data/control load #615
Description
At 3kqps and control plane updating vm_id for metadata_exchange v8 raw wasm every 2s using ECDS, I'm seeing a crash about 90 minutes in:
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x55c100000000
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: e275bca85d3412681b18275274d61c55573db731/1.15.0/Modified/DEBUG/BoringSSL
[2020-08-11 15:51:26.502][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x55c10bc7433c]
[2020-08-11 15:51:26.502][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f55d3e4c8a0]
[2020-08-11 15:51:26.560][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: proxy_wasm::(anonymous namespace)::V8::getModuleFunctionImpl<>()::{lambda()#1}::operator()() [0x55c109f73ca5]
[2020-08-11 15:51:26.617][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::_Function_handler<>::_M_invoke() [0x55c109f73a62]
[2020-08-11 15:51:26.675][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::function<>::operator()() [0x55c109f4eda0]
[2020-08-11 15:51:26.732][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: proxy_wasm::ContextBase::onLog() [0x55c109f4b0bc]
[2020-08-11 15:51:26.790][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Extensions::Common::Wasm::Context::log() [0x55c109be57d3]
[2020-08-11 15:51:26.847][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b75950d]
[2020-08-11 15:51:26.905][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b759e20]
[2020-08-11 15:51:26.962][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b759f19]
[2020-08-11 15:51:27.019][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::default_delete<>::operator()() [0x55c108c907df]
[2020-08-11 15:51:27.077][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::unique_ptr<>::reset() [0x55c109c2a5c6]
[2020-08-11 15:51:27.134][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Event::DispatcherImpl::clearDeferredDeleteList() [0x55c10b179bb0]
[2020-08-11 15:51:27.192][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Event::DispatcherImpl::DispatcherImpl()::$_0::operator()() [0x55c10b17c0f9]
[2020-08-11 15:51:27.250][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::_Function_handler<>::_M_invoke() [0x55c10b17bfad]
[2020-08-11 15:51:27.307][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::function<>::operator()() [0x55c108704bce]
[2020-08-11 15:51:27.365][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::operator()() [0x55c10b1b0cb5]
[2020-08-11 15:51:27.422][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::__invoke() [0x55c10b1b0c79]
[2020-08-11 15:51:27.480][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: event_process_active_single_queue [0x55c10bb79be9]
[2020-08-11 15:51:27.537][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: event_process_active [0x55c10bb7415b]
[2020-08-11 15:51:27.595][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: event_base_loop [0x55c10bb72fce]
[2020-08-11 15:51:27.652][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: Envoy::Event::LibeventScheduler::run() [0x55c10b1aeaac]
[2020-08-11 15:51:27.709][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: Envoy::Event::DispatcherImpl::run() [0x55c10b17bb28]
[2020-08-11 15:51:27.766][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: Envoy::Server::WorkerImpl::threadRoutine() [0x55c10b158219]
[2020-08-11 15:51:27.823][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: Envoy::Server::WorkerImpl::start()::$_4::operator()() [0x55c10b15994f]
[2020-08-11 15:51:27.880][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #25: std::_Function_handler<>::_M_invoke() [0x55c10b1597fd]
[2020-08-11 15:51:27.880][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: std::function<>::operator()() [0x55c108704bce]
[2020-08-11 15:51:27.937][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #27: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x55c10bcd2a70]
[2020-08-11 15:51:27.994][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #28: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x55c10bcd2a45]
[2020-08-11 15:51:27.994][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #29: start_thread [0x7f55d3e416db]
Segmentation fault
This does not appear to be OOM since I was monitoring the physical bytes (hovering around 100MB).