Segfault in thrift proxy #6496
Description
Title: We're seeing a recurring segfault in the thrift proxy, stacktrace below. We're actively working on diagnosing and reproducing.
Description:
Segfaulting occurs some time after starting (up to about 15 minutes) when running Thrift for ingress traffic. We're running from commit 7ed6d21 (April 4th).
Repro steps:
Working on it. @fishcakez @rgs1 and I are actively debugging.
Config:
Listeners:
[
{
"@type": "type.googleapis.com/envoy.api.v2.Listener",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 8000
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"route_config": {
"name": "routes_service",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "local_service"
}
}
]
},
"stat_prefix": "ingress_service",
"thrift_filters": [
{
"name": "envoy.filters.thrift.router",
"typed_config": {}
}
]
}
}
],
"tls_context": {
"common_tls_context": {
"tls_certificate_sds_secret_configs": [
{
"name": "certificate_chain",
"sds_config": {
"api_config_source": {
"api_type": "GRPC",
"grpc_services": [
{
"envoy_grpc": {
"cluster_name": "local_sds"
}
}
]
}
}
}
],
"validation_context_sds_secret_config": {
"name": "certificate_authority",
"sds_config": {
"api_config_source": {
"api_type": "GRPC",
"grpc_services": [
{
"envoy_grpc": {
"cluster_name": "local_sds"
}
}
]
}
}
}
},
"require_client_certificate": true
}
}
],
"name": "ingress_service"
}
]
cluster config
[
{
"@type": "type.googleapis.com/envoy.api.v2.Cluster",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 16384,
"max_pending_requests": 16384,
"max_requests": 16384,
"priority": "DEFAULT"
}
]
},
"connect_timeout": "0.25s",
"lb_policy": "ROUND_ROBIN",
"load_assignment": {
"cluster_name": "local_service",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8020
}
}
}
}
]
}
]
},
"name": "local_service",
"type": "STATIC"
}
]
Call Stack + Last Logs:
...
[2019-04-05 23:04:16.193][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 2 bytes available
[2019-04-05 23:04:16.193][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 1 bytes available
[2019-04-05 23:04:16.193][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldEnd, 1 bytes available
[2019-04-05 23:04:16.195][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state FieldBegin, 1 bytes available
[2019-04-05 23:04:16.197][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state StructEnd, 0 bytes available
[2019-04-05 23:04:16.197][19][trace][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:338] thrift: state MessageEnd, 0 bytes available
[2019-04-05 23:04:16.197][19][debug][thrift] [external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:446] thrift: framed transport ended
[2019-04-05 23:04:16.199][19][trace][connection] [external/envoy/source/common/network/connection_impl.cc:376] [C116] writing 239 bytes, end_stream false
[2019-04-05 23:04:16.199][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x0
[2019-04-05 23:04:16.201][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-04-05 23:04:16.201][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7f9a1d6b5390] ??:0
[2019-04-05 23:04:16.205][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Extensions::NetworkFilters::ThriftProxy::ConnectionManager::ResponseDecoder::transportEnd() [0x736533] external/envoy/source/extensions/filters/network/thrift_proxy/conn_manager.cc:221
[2019-04-05 23:04:16.207][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Extensions::NetworkFilters::ThriftProxy::Decoder::onData() [0x73ab1a] external/envoy/source/extensions/filters/network/thrift_proxy/decoder.cc:447
[2019-04-05 23:04:16.209][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Extensions::NetworkFilters::ThriftProxy::ConnectionManager::ResponseDecoder::onData() [0x732d7e] external/envoy/source/extensions/filters/network/thrift_proxy/conn_manager.cc:176
[2019-04-05 23:04:16.213][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::NetworkFilters::ThriftProxy::ConnectionManager::ActiveRpc::upstreamData() [0x737eab] external/envoy/source/extensions/filters/network/thrift_proxy/conn_manager.cc:636
[2019-04-05 23:04:16.215][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Extensions::NetworkFilters::ThriftProxy::Router::Router::onUpstreamData() [0x72c8af] external/envoy/source/extensions/filters/network/thrift_proxy/router/router_impl.cc:309
[2019-04-05 23:04:16.217][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Tcp::ConnPoolImpl::ActiveConn::onUpstreamData() [0x9bc849] external/envoy/source/common/tcp/conn_pool.cc:426
[2019-04-05 23:04:16.219][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Tcp::ConnPoolImpl::ConnReadFilter::onData() [0x9bc900] bazel-out/k8-opt/bin/external/envoy/source/common/tcp/_virtual_includes/conn_pool_lib/common/tcp/conn_pool.h:86
[2019-04-05 23:04:16.223][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Network::FilterManagerImpl::onContinueReading() [0x8aa5cc] external/envoy/source/common/network/filter_manager_impl.cc:57
[2019-04-05 23:04:16.226][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Network::ConnectionImpl::onReadReady() [0x8a6e7a] external/envoy/source/common/network/connection_impl.cc:263
[2019-04-05 23:04:16.229][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Network::ConnectionImpl::onFileEvent() [0x8a758a] external/envoy/source/common/network/connection_impl.cc:473
[2019-04-05 23:04:16.231][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN() [0x8a12da] external/envoy/source/common/event/file_event_impl.cc:65
[2019-04-05 23:04:16.233][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: event_process_active_single_queue.isra.31 [0xbd0fb9] event.c:?
[2019-04-05 23:04:16.235][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: event_process_active [0xbd14ef] event.c:?
[2019-04-05 23:04:16.656][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: event_base_loop [0xbd30f8] ??:?
[2019-04-05 23:04:17.052][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Event::DispatcherImpl::run() [0x8a097a] external/envoy/source/common/event/dispatcher_impl.cc:168
[2019-04-05 23:04:17.453][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Server::WorkerImpl::threadRoutine() [0x89b89e] bazel-out/k8-opt/bin/external/envoy/source/common/common/_virtual_includes/minimal_logger_lib/common/common/logger.h:274
[2019-04-05 23:04:17.853][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0xdab295] external/envoy/source/common/common/posix/thread_impl.cc:40
[2019-04-05 23:04:18.256][19][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: start_thread [0x7f9a1d6ab6ba] ??:0