Skip to content

upstream: crash due to regression in #6765 #6806

@mattklein123

Description

@mattklein123
Backtrace:
#0  Envoy::Upstream::HealthCheckerImplBase::ActiveHealthCheckSession::handleFailure (this=0x7108d10, type=<optimized out>) at external/envoy/source/common/upstream/health_checker_base_impl.cc:313
#0  Envoy::Upstream::HealthCheckerImplBase::ActiveHealthCheckSession::handleFailure (this=0x7108d10, type=<optimized out>) at external/envoy/source/common/upstream/health_checker_base_impl.cc:313
#1  0x000000000073f823 in Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::onResponseComplete (this=0x7108d10) at external/envoy/source/common/upstream/health_checker_impl.cc:263
#2  0x000000000074c77c in Envoy::Http::StreamDecoderWrapper::decodeHeaders(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&, bool) (this=0xf65e0a0, headers=<unknown type in /usr/sbin/envoy, CU 0x829dc84, DIE 0x8347c89>, end_stream=<optimized out>) at bazel-out/k8-opt/bin/external/envoy/source/common/http/_virtual_includes/codec_wrappers_lib/common/http/codec_wrappers.h:23
#3  0x00000000007c8b26 in Envoy::Http::Http1::ClientConnectionImpl::onMessageComplete (this=<optimized out>) at external/envoy/source/common/http/http1/codec_impl.cc:759
#4  0x00000000007c6639 in Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase (this=0x7131088) at external/envoy/source/common/http/http1/codec_impl.cc:475
#5  0x00000000007c9fdd in operator() (this=<optimized out>, parser=<optimized out>) at external/envoy/source/common/http/http1/codec_impl.cc:310
#6  Envoy::Http::Http1::ConnectionImpl::$_7::__invoke (parser=<optimized out>) at external/envoy/source/common/http/http1/codec_impl.cc:309
#7  0x0000000000984ebb in http_parser_execute ()
#8  0x00000000007c5fa4 in Envoy::Http::Http1::ConnectionImpl::dispatchSlice (this=0x7131088, slice=0x7ffdad5ce7e8 "\001", len=1934) at external/envoy/source/common/http/http1/codec_impl.cc:396
#9  0x00000000007c5d9f in Envoy::Http::Http1::ConnectionImpl::dispatch (this=0x7131088, data=...) at external/envoy/source/common/http/http1/codec_impl.cc:381
#10 0x000000000074bbc6 in Envoy::Http::CodecClient::onData (this=0x5910580, data=...) at external/envoy/source/common/http/codec_client.cc:116
#11 0x000000000074c64d in Envoy::Http::CodecClient::CodecReadFilter::onData (this=<optimized out>, data=...) at bazel-out/k8-opt/bin/external/envoy/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:167
#12 0x000000000068231a in onContinueReading (this=0x6b8fc20, filter=0x0) at external/envoy/source/common/network/filter_manager_impl.cc:56
#13 Envoy::Network::FilterManagerImpl::onRead (this=0x6b8fc20) at external/envoy/source/common/network/filter_manager_impl.cc:66
#14 0x000000000067ef7e in onRead (this=<optimized out>, read_buffer_size=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:272
#15 Envoy::Network::ConnectionImpl::onReadReady (this=0x6b8fc00) at external/envoy/source/common/network/connection_impl.cc:506
#16 0x000000000067ea51 in Envoy::Network::ConnectionImpl::onFileEvent (this=0x6b8fc00, events=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:482
#17 0x0000000000679a35 in operator() (this=0x0, __args=3) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/std_function.h:706
#18 operator() (this=<optimized out>, what=<optimized out>, arg=0x7ffdad5ce7e8) at external/envoy/source/common/event/file_event_impl.cc:65
#19 Envoy::Event::FileEventImpl::assignEvents(unsigned int)::$_0::__invoke(int, short, void*) (what=<optimized out>, arg=0x7ffdad5ce7e8) at external/envoy/source/common/event/file_event_impl.cc:49
#20 0x000000000097609d in event_process_active_single_queue ()
#21 0x0000000000974640 in event_base_loop ()
#22 0x00000000006787ed in Envoy::Event::DispatcherImpl::run (this=<optimized out>, type=Envoy::Event::Dispatcher::Block) at external/envoy/source/common/event/dispatcher_impl.cc:178
#23 0x000000000062ba85 in Envoy::Server::InstanceImpl::run (this=0x3968500) at external/envoy/source/server/server.cc:480
#24 0x000000000045b4bc in Envoy::MainCommonBase::run (this=0x39bc6e8) at external/envoy/source/exe/main_common.cc:124
#25 0x000000000045a354 in run (this=0x39bc600) at bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:89
#26 main (argc=17, argv=<optimized out>) at external/envoy/source/exe/main.cc:39

We have an issue where a host can get removed, which triggers further removal callbacks and kills the health check session, and then crash trying to access the session after the removal.

I'm going to figure out how to fix this but if anyone hits this and wants it reverted LMK and I will revert the original commit. cc @snowp

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions