Description:

See the title.

The ActiveStream is being destroyed by a deferred delete and it is occurring after the filter chain has been destroyed. It looks like the UpstreamRequest destructor is calling the upstream_logs_ ->log() call which in turn is calling the log() API in the filter for a filter chain which is already destroyed. While it is easy enough to detect this and avoid using state which is no longer valid I am wondering if this is the intended design? I would think that it would be best to avoid calls to filters on a filter chain which has already been destoryed.

[optional Relevant Links:]

envoyproxy/envoy-wasm#615

This occurs during rapid reconfiguration of Wasm filters.

Repro steps:

Add a Wasm filter which which includes a log() handler (e.g. HTTP filter) and reconfigure the filter chains rapidly (every 2 seconds) while under load.

[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x560500000000
[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: e275bca85d3412681b18275274d61c55573db731/1.15.0/Modified/DEBUG/BoringSSL
[2020-08-11 21:13:00.478][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x560523a9f33c]
[2020-08-11 21:13:00.479][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f19533f68a0]
[2020-08-11 21:13:00.536][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: proxy_wasm::(anonymous namespace)::V8::getModuleFunctionImpl<>()::{lambda()#1}::operator()() [0x560521d9eca5]
[2020-08-11 21:13:00.594][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::_Function_handler<>::_M_invoke() [0x560521d9ea62]
[2020-08-11 21:13:00.652][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::function<>::operator()() [0x560521d79da0]
[2020-08-11 21:13:00.709][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: proxy_wasm::ContextBase::onLog() [0x560521d760bc]
[2020-08-11 21:13:00.767][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Extensions::Common::Wasm::Context::log() [0x560521a107d3]
[2020-08-11 21:13:00.825][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x56052358450d]
[2020-08-11 21:13:00.883][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x560523584e20]
[2020-08-11 21:13:00.941][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x560523584f19]
[2020-08-11 21:13:00.999][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::default_delete<>::operator()() [0x560520abb7df]
[2020-08-11 21:13:01.056][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::unique_ptr<>::reset() [0x560521a555c6]
[2020-08-11 21:13:01.114][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Event::DispatcherImpl::clearDeferredDeleteList() [0x560522fa4bb0]
[2020-08-11 21:13:01.173][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Event::DispatcherImpl::DispatcherImpl()::$_0::operator()() [0x560522fa70f9]
[2020-08-11 21:13:01.232][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::_Function_handler<>::_M_invoke() [0x560522fa6fad]
[2020-08-11 21:13:01.290][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::function<>::operator()() [0x56052052fbce]
[2020-08-11 21:13:01.347][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::operator()() [0x560522fdbcb5]