-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Closed
Labels
Description
Given a cluster performing http health checks, change the priority of an existing LocalityLbEndpoints containing at least one healthy host from priority >0 to 0.
This results in:
[source/common/upstream/health_checker_base_impl.cc:265] assert failure: interval_timer_ == nullptr && timeout_timer_ == nullptr.
Discussion on the security group concluded that there's no security issue here, so I'm re-raising it on github for tracking.
Call Stack:
[2020-05-27 13:18:39.912][592380][critical][assert] [source/common/upstream/health_checker_base_impl.cc:265] assert failure: interval_timer_ == nullptr && timeout_timer_ == nullptr.
[2020-05-27 13:18:39.912][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x644159a6000909fc
[2020-05-27 13:18:39.912][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-05-27 13:18:39.912][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: c8f330c1f99ee5d89048d52ac98faab2539d5a52/1.15.0-dev/Clean/DEBUG/BoringSSL
[2020-05-27 13:18:39.964][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x5562d0a459cc]
[2020-05-27 13:18:39.964][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f1d9610f3c0]
[2020-05-27 13:18:40.014][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::~HttpActiveHealthCheckSession() [0x5562d02c32c6]
[2020-05-27 13:18:40.062][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::~HttpActiveHealthCheckSession() [0x5562d02c3383]
[2020-05-27 13:18:40.114][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::~HttpActiveHealthCheckSession() [0x5562d02c341c]
[2020-05-27 13:18:40.168][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: std::__1::default_delete<>::operator()() [0x5562d02ec72f]
[2020-05-27 13:18:40.219][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: std::__1::unique_ptr<>::reset() [0x5562d02ec6af]
[2020-05-27 13:18:40.268][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: std::__1::unique_ptr<>::operator=() [0x5562d02e721d]
[2020-05-27 13:18:40.318][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Upstream::HealthCheckerImplBase::addHosts() [0x5562d02d8857]
[2020-05-27 13:18:40.366][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Upstream::HealthCheckerImplBase::onClusterMemberUpdate() [0x5562d02d8a3e]
[2020-05-27 13:18:40.417][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: Envoy::Upstream::HealthCheckerImplBase::HealthCheckerImplBase()::$_5::operator()() [0x5562d02dc0a8]
[2020-05-27 13:18:40.465][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::__1::__invoke<>() [0x5562d02dc067]
[2020-05-27 13:18:40.517][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x5562d02dbff7]
[2020-05-27 13:18:40.564][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: std::__1::__function::__alloc_func<>::operator()() [0x5562d02dbfa7]
[2020-05-27 13:18:40.612][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::__1::__function::__func<>::operator()() [0x5562d02db0d8]
[2020-05-27 13:18:40.659][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::__1::__function::__value_func<>::operator()() [0x5562cff73fca]
[2020-05-27 13:18:40.708][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: std::__1::function<>::operator()() [0x5562cff73ea5]
[2020-05-27 13:18:40.757][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: Envoy::Common::CallbackManager<>::runCallbacks() [0x5562cff73d7d]
[2020-05-27 13:18:40.805][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: Envoy::Upstream::PrioritySetImpl::runUpdateCallbacks() [0x5562cff6f68e]
[2020-05-27 13:18:40.851][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: Envoy::Upstream::PrioritySetImpl::batchHostUpdate() [0x5562d01d5c6c]
[2020-05-27 13:18:40.899][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: Envoy::Upstream::EdsClusterImpl::onConfigUpdate() [0x5562d0229cb2]
[2020-05-27 13:18:40.949][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x5562d025f948]
[2020-05-27 13:18:41.000][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x5562d0263b9d]
[2020-05-27 13:18:41.047][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: Envoy::Config::GrpcStream<>::onReceiveMessage() [0x5562cff0f8b1]
[2020-05-27 13:18:41.096][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x5562cff0f699]
[2020-05-27 13:18:41.146][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #25: Envoy::Grpc::AsyncStreamImpl::onData() [0x5562d030ef73]
[2020-05-27 13:18:41.196][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: Envoy::Http::AsyncStreamImpl::encodeData() [0x5562d031b1fb]
[2020-05-27 13:18:41.244][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #27: Envoy::Router::Filter::onUpstreamData() [0x5562d03372eb]
[2020-05-27 13:18:41.297][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #28: Envoy::Router::UpstreamRequest::decodeData() [0x5562d035d494]
[2020-05-27 13:18:41.345][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #29: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x5562d01b52dc]
[2020-05-27 13:18:41.397][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #30: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x5562d04f86cb]
[2020-05-27 13:18:41.445][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #31: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_14::operator()() [0x5562d04fde28]
[2020-05-27 13:18:41.497][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #32: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_14::__invoke() [0x5562d04fddf5]
[2020-05-27 13:18:41.544][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #33: session_call_on_frame_received [0x5562cfa08622]
[2020-05-27 13:18:41.594][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #34: nghttp2_session_on_data_received [0x5562cfa0a0ce]
[2020-05-27 13:18:41.642][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #35: session_process_data_frame [0x5562cfa0eb97]
[2020-05-27 13:18:41.693][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #36: nghttp2_session_mem_recv [0x5562cfa0c98c]
[2020-05-27 13:18:41.741][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #37: Envoy::Http::Http2::ConnectionImpl::innerDispatch() [0x5562d04f7051]
[2020-05-27 13:18:41.792][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #38: Envoy::Http::Http2::ConnectionImpl::dispatch()::$_5::operator()() [0x5562d05039ca]
[2020-05-27 13:18:41.838][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #39: std::__1::__invoke<>() [0x5562d050396b]
[2020-05-27 13:18:41.887][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #40: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x5562d05038eb]
[2020-05-27 13:18:41.937][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #41: std::__1::__function::__alloc_func<>::operator()() [0x5562d050388b]
[2020-05-27 13:18:41.981][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #42: std::__1::__function::__func<>::operator()() [0x5562d05029ad]
[2020-05-27 13:18:42.029][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #43: std::__1::__function::__value_func<>::operator()() [0x5562d072107b]
[2020-05-27 13:18:42.077][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #44: std::__1::function<>::operator()() [0x5562d071ce86]
[2020-05-27 13:18:42.123][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #45: Envoy::Http::Utility::exceptionToStatus() [0x5562d0711ea8]
[2020-05-27 13:18:42.169][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #46: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x5562d04f6cd2]
[2020-05-27 13:18:42.214][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #47: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x5562d04f6dab]
[2020-05-27 13:18:42.260][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #48: Envoy::Http::CodecClient::onData() [0x5562d02ee87b]
[2020-05-27 13:18:42.306][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #49: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x5562d02f1f47]
[2020-05-27 13:18:42.355][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #50: Envoy::Network::FilterManagerImpl::onContinueReading() [0x5562cfded8d5]
[2020-05-27 13:18:42.404][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #51: Envoy::Network::FilterManagerImpl::onRead() [0x5562cfdedad7]
[2020-05-27 13:18:42.448][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #52: Envoy::Network::ConnectionImpl::onRead() [0x5562cfdd94c8]
[2020-05-27 13:18:42.495][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #53: Envoy::Network::ConnectionImpl::onReadReady() [0x5562cfddcd99]
[2020-05-27 13:18:42.543][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #54: Envoy::Network::ConnectionImpl::onFileEvent() [0x5562cfddb97c]
[2020-05-27 13:18:42.590][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #55: Envoy::Network::ConnectionImpl::ConnectionImpl()::$_4::operator()() [0x5562cfde4d0e]
[2020-05-27 13:18:42.636][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #56: std::__1::__invoke<>() [0x5562cfde4cd1]
[2020-05-27 13:18:42.685][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #57: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x5562cfde4c72]
[2020-05-27 13:18:42.734][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #58: std::__1::__function::__alloc_func<>::operator()() [0x5562cfde4c32]
[2020-05-27 13:18:42.779][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #59: std::__1::__function::__func<>::operator()() [0x5562cfde3d73]
[2020-05-27 13:18:42.826][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #60: std::__1::__function::__value_func<>::operator()() [0x5562cfdce7bd]
[2020-05-27 13:18:42.873][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #61: std::__1::function<>::operator()() [0x5562cfdce75f]
[2020-05-27 13:18:42.922][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #62: Envoy::Event::FileEventImpl::assignEvents()::$_0::operator()() [0x5562cfdce442]
[2020-05-27 13:18:42.971][592380][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #63: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x5562cfdce3a6]
AsyncClient 0x5562d407a900, stream_id_: 8013577971860007246
&stream_info_:
StreamInfoImpl 0x5562d407aab0, protocol_: 1, response_code_: null, response_code_details_: via_upstream, health_check_request_: 0, route_name_:
[1] 592380 abort (core dumped) ~/IdeaProjects/envoy/build_debug/envoy -c config/sotw.yaml
Reactions are currently unavailable