Envoy::Router::PrefixRouteEntryImpl::matches() Caught Segmentation fault, suspect faulting address #6348
Description
Bug Template
Title: One line description
Description:
I try to run Envoy on AWS EC2 c5n.9xlarge with Gloo on Kubernetes,
I run it on production to serve 2,000,000+ RPM (http+https)
after running it for 10min, it crashed.
I have 2 listeners
- HTTP
- HTTPS
both are configured with the following routers
- 0: /path1 -> static host
- 1: /path.php -> static host
- 2: / -> dynamic to Kubernetes service
Envoy version: 1.10.0-dev (79dca84)
Repro steps:
Configure Envoy with 2 listeners and use similar routing like above and try to send it 200000 RPM/m for 10min+
Call Stack:
[2019-03-21 17:57:40.622][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2019-03-21 17:57:40.622][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<86> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2019-03-21 17:57:40.625][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #0 0xaf1c30 Envoy::Router::PrefixRouteEntryImpl::matches()
[2019-03-21 17:57:40.627][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #1 0xaf1cc1 Envoy::Router::PrefixRouteEntryImpl::matches()
[2019-03-21 17:57:40.629][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #2 0xaf3cf8 Envoy::Router::VirtualHostImpl::getRouteFromEntries()
[2019-03-21 17:57:40.631][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #3 0xaf7974 Envoy::Router::ConfigImpl::route()
[2019-03-21 17:57:40.633][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #4 0x9cef46 Envoy::Http::ConnectionManagerImpl::ActiveStream::refreshCachedRoute()
[2019-03-21 17:57:40.635][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #5 0x9d2d2e Envoy::Http::ConnectionManagerImpl::ActiveStreamFilterBase::route()
[2019-03-21 17:57:40.637][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #6 0x4793d9 Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::checkResponseActive()
[2019-03-21 17:57:40.639][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #7 0x478fdb Envoy::Extensions::HttpFilters::Transformation::TransformationFilter::encodeHeaders()
[2019-03-21 17:57:40.641][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #8 0x9d096e Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders()
[2019-03-21 17:57:40.643][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #9 0xb4feb2 Envoy::Http::Utility::sendLocalReply()
[2019-03-21 17:57:40.646][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #10 0x9cd9fc Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply()
[2019-03-21 17:57:40.648][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #11 0x9cd861 Envoy::Http::ConnectionManagerImpl::ActiveStream::onIdleTimeout()
[2019-03-21 17:57:40.650][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #12 0xb97d67 event_process_active_single_queue.isra.29
[2019-03-21 17:57:40.652][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #13 0xb9845e event_process_active
[2019-03-21 17:57:40.654][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #14 0xb9b1e7 event_base_loop
[2019-03-21 17:57:40.656][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #15 0x80bb42 Envoy::Event::DispatcherImpl::run()
[2019-03-21 17:57:40.658][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #16 0x80723e Envoy::Server::WorkerImpl::threadRoutine()
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<86> #17 0xe0b9a4 Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke()
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<86> obj</usr/glibc-compat/lib/libpthread.so.0>
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<86> #18 0x7fa0543d3d2b (unknown)
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<86> obj</usr/glibc-compat/lib/libc.so.6>
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<86> #19 0x7fa053d83e3e (unknown)
[2019-03-21 17:57:40.660][86][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 86