Crash when load balancer rejects host_set change due to health checks #41102
Description
If a load balancer returns an error from it's priority_set update callback, and the priority set is updated due to a healthcheck status change, Envoy crashes with an uncaught exception.
The only configuration I'm aware of that triggers this is endpoints with load_balancing_weight's that sum to more than UINT32_MAX, and using a thread aware load balancer (maglev or ring_hash).
The crash occurs on the main thread:
frame #0: 0x00000001006bcbc8 eds_integration_test`Envoy::Upstream::(anonymous namespace)::normalizeHostWeights(hosts=size=2, normalized_locality_weight=1, normalized_host_weights=size=0, min_normalized_weight=0x00000001700adbf0, max_normalized_weight=0x00000001700adbe8) at thread_aware_lb_impl.cc:26:14
frame #1: 0x00000001006b9f08 eds_integration_test`Envoy::Upstream::(anonymous namespace)::normalizeWeights(host_set=0x0000000151f14290, in_panic=false, normalized_host_weights=size=0, min_normalized_weight=0x00000001700adbf0, max_normalized_weight=0x00000001700adbe8, locality_weighted_balancing=false) at thread_aware_lb_impl.cc:87:5
frame #2: 0x00000001006b982c eds_integration_test`Envoy::Upstream::ThreadAwareLoadBalancerBase::refresh(this=0x0000000151f14f00) at thread_aware_lb_impl.cc:164:27
frame #3: 0x00000001006c0c88 eds_integration_test`Envoy::Upstream::ThreadAwareLoadBalancerBase::initialize()::$_0::operator()(this=0x0000600000c3fd08, (null)=0, (null)=size=0, (null)=size=0) const at thread_aware_lb_impl.cc:139:87
frame #9: 0x00000001014cf144 eds_integration_test`std::__1::function<absl::lts_20250512::Status (unsigned int, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&)>::operator()(this= Lambda in File thread_aware_lb_impl.cc at Line 139, __arg=0, __arg=size=0, __arg=size=0) const at function.h:989:10
frame #10: 0x00000001014c6b30 eds_integration_test`Envoy::Common::CallbackManager<unsigned int, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&>::runCallbacks(this=0x0000000151f11b80, args=0, args=size=0, args=size=0) at callback_impl.h:52:7
frame #11: 0x0000000102c0f02c eds_integration_test`Envoy::Upstream::PrioritySetImpl::runReferenceUpdateCallbacks(this=0x0000000151f11b10, priority=0, hosts_added=size=0, hosts_removed=size=0) at upstream_impl.h:754:39
frame #12: 0x0000000102c4aa2c eds_integration_test`Envoy::Upstream::PrioritySetImpl::getOrCreateHostSet(unsigned int, std::__1::optional<bool>, std::__1::optional<unsigned int>)::$_0::operator()(this=0x0000600000c3f948, priority=0, hosts_added=size=0, hosts_removed=size=0) const at upstream_impl.cc:905:20
frame #18: 0x00000001014cf144 eds_integration_test`std::__1::function<absl::lts_20250512::Status (unsigned int, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&)>::operator()(this=0x0000600000c3f940, __arg=0, __arg=size=0, __arg=size=0) const at function.h:989:10
frame #19: 0x00000001014c6b30 eds_integration_test`Envoy::Common::CallbackManager<unsigned int, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&, std::__1::vector<std::__1::shared_ptr<Envoy::Upstream::Host>, std::__1::allocator<std::__1::shared_ptr<Envoy::Upstream::Host>>> const&>::runCallbacks(this=0x0000000151f14328, args=0, args=size=0, args=size=0) at callback_impl.h:52:7
frame #20: 0x0000000102c0ee68 eds_integration_test`Envoy::Upstream::HostSetImpl::runUpdateCallbacks(this=0x0000000151f14290, hosts_added=size=0, hosts_removed=size=0) at upstream_impl.h:629:37
frame #21: 0x0000000102bf27e0 eds_integration_test`Envoy::Upstream::HostSetImpl::updateHosts(this=0x0000000151f14290, update_hosts_params=0x00000001700aeaa8, locality_weights=nullptr, hosts_added=size=0, hosts_removed=size=0, seed=12116778962163545064, weighted_priority_health= Has Value=false , overprovisioning_factor= Has Value=false ) at upstream_impl.cc:761:3
frame #22: 0x0000000102bf5414 eds_integration_test`Envoy::Upstream::PrioritySetImpl::updateHosts(this=0x0000000151f11b10, priority=0, update_hosts_params=0x00000001700aeaa8, locality_weights=nullptr, hosts_added=size=0, hosts_removed=size=0, seed=12116778962163545064, weighted_priority_health= Has Value=false , overprovisioning_factor= Has Value=false , cross_priority_host_map=nullptr) at upstream_impl.cc:928:9
frame #23: 0x0000000102bf65b0 eds_integration_test`Envoy::Upstream::MainPrioritySetImpl::updateHosts(this=0x0000000151f11b10, priority=0, update_hosts_params=0x00000001700aeaa8, locality_weights=size=1 strong=4 weak=1, hosts_added=size=0, hosts_removed=size=0, seed=12116778962163545064, weighted_priority_health= Has Value=false , overprovisioning_factor= Has Value=false , cross_priority_host_map=nullptr) at upstream_impl.cc:981:20
frame #24: 0x000000010051f020 eds_integration_test`Envoy::Upstream::EdsClusterImpl::reloadHealthyHostsHelper(this=0x0000000151f119f0, host=std::__1::shared_ptr<Envoy::Upstream::Host>::element_type @ 0x0000000151e71770 strong=14 weak=3) at eds.cc:407:19
frame #25: 0x0000000102c0567c eds_integration_test`Envoy::Upstream::ClusterImplBase::reloadHealthyHosts(this=0x0000000151f119f0, host=std::__1::shared_ptr<Envoy::Upstream::Host>::element_type @ 0x0000000151e71770 strong=14 weak=3) at upstream_impl.cc:1931:3
* frame #26: 0x0000000102c781e0 eds_integration_test`Envoy::Upstream::ClusterImplBase::setHealthChecker(std::__1::shared_ptr<Envoy::Upstream::HealthChecker> const&)::$_1::operator()(this=0x00006000021b7ac8, host=std::__1::shared_ptr<Envoy::Upstream::Host>::element_type @ 0x0000000151e71770 strong=14 weak=3, changed_state=Changed, (null)=Healthy) const at upstream_impl.cc:1906:11
frame #32: 0x000000010328f77c eds_integration_test`std::__1::function<void (std::__1::shared_ptr<Envoy::Upstream::Host> const&, Envoy::Upstream::HealthTransition, Envoy::Upstream::HealthState)>::operator()(this= Lambda in File upstream_impl.cc at Line 1902, __arg=std::__1::shared_ptr<Envoy::Upstream::Host>::element_type @ 0x0000000151e71770 strong=14 weak=3, __arg=Changed, __arg=Healthy) const at function.h:989:10
frame #33: 0x000000010328f674 eds_integration_test`Envoy::Upstream::HealthCheckerImplBase::runCallbacks(this=0x0000000152831018, host=std::__1::shared_ptr<Envoy::Upstream::Host>::element_type @ 0x0000000151e71770 strong=14 weak=3, changed_state=Changed, current_check_result=Healthy) at health_checker_base_impl.cc:193:5
frame #34: 0x0000000103290768 eds_integration_test`Envoy::Upstream::HealthCheckerImplBase::ActiveHealthCheckSession::handleSuccess(this=0x0000600001a24e40, degraded=false) at health_checker_base_impl.cc:343:11
frame #35: 0x0000000102d65ea0 eds_integration_test`Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::onResponseComplete(this=0x0000600001a24e40) at health_checker_impl.cc:424:5
frame #36: 0x0000000102d660e4 eds_integration_test`Envoy::Upstream::HttpHealthCheckerImpl::HttpActiveHealthCheckSession::decodeData(this=0x0000600001a24e40, data=0x00000001700af380, end_stream=true) at health_checker_impl.cc:254:5
frame #37: 0x00000001024569c8 eds_integration_test`Envoy::Http::ResponseDecoderWrapper::decodeData(this=0x0000600000b3f780, data=0x00000001700af380, end_stream=true) at codec_wrappers.h:32:12
frame #38: 0x0000000102db6ed0 eds_integration_test`Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase(this=0x0000000152154000) at codec_impl.cc:1610:26
frame #39: 0x0000000102daa0e8 eds_integration_test`Envoy::Http::Http1::ConnectionImpl::onMessageCompleteImpl(this=0x0000000152154000) at codec_impl.cc:976:10
frame #40: 0x0000000102da973c eds_integration_test`Envoy::Http::Http1::ConnectionImpl::onMessageComplete(this=0x0000000152154000) at codec_impl.cc:771:38
frame #41: 0x0000000102dd6014 eds_integration_test`Envoy::Http::Http1::BalsaParser::MessageDone(this=0x0000000151e72330) at balsa_parser.cc:372:40
frame #42: 0x0000000102ddf71c eds_integration_test`quiche::BalsaFrame::ProcessInput(this=0x0000000151e72340, input="a\r\naaaaaaaaaa\r\n0\r\n\r\n", size=20) at balsa_frame.cc:1399:23
frame #43: 0x0000000102dd4018 eds_integration_test`Envoy::Http::Http1::BalsaParser::execute(this=0x0000000151e72330, slice="a\r\naaaaaaaaaa\r\n0\r\n\r\n", len=20) at balsa_parser.cc:232:18
frame #44: 0x0000000102da553c eds_integration_test`Envoy::Http::Http1::ConnectionImpl::dispatchSlice(this=0x0000000152154000, slice="a\r\naaaaaaaaaa\r\n0\r\n\r\n", len=20) at codec_impl.cc:709:33
frame #45: 0x0000000102da4130 eds_integration_test`Envoy::Http::Http1::ConnectionImpl::dispatch(this=0x0000000152154000, data=0x00000001520c2400) at codec_impl.cc:672:30
frame #46: 0x0000000102da33d4 eds_integration_test`Envoy::Http::Http1::ClientConnectionImpl::dispatch(this=0x0000000152154000, data=0x00000001520c2400) at codec_impl.cc:637:41
frame #47: 0x0000000102d92ffc eds_integration_test`Envoy::Http::CodecClient::onData(this=0x0000600001d0e130, data=0x00000001520c2400) at codec_client.cc:168:33
frame #48: 0x0000000102d961c0 eds_integration_test`Envoy::Http::CodecClient::CodecReadFilter::onData(this=0x0000600002d26810, data=0x00000001520c2400, end_stream=false) at codec_client.h:211:15
frame #49: 0x0000000103af6b5c eds_integration_test`Envoy::Network::FilterManagerImpl::onContinueReading(this=0x0000000152076290, filter=0x0000000000000000, buffer_source=0x0000000152076200) at filter_manager_impl.cc:92:48
frame #50: 0x0000000103af718c eds_integration_test`Envoy::Network::FilterManagerImpl::onRead(this=0x0000000152076290) at filter_manager_impl.cc:102:3
frame #51: 0x0000000103ad0ed0 eds_integration_test`Envoy::Network::ConnectionImpl::onRead(this=0x0000000152076200, read_buffer_size=67) at connection_impl.cc:431:19
frame #52: 0x0000000103adae7c eds_integration_test`Envoy::Network::ConnectionImpl::onReadReady(this=0x0000000152076200) at connection_impl.cc:775:5
frame #53: 0x0000000103ad6a44 eds_integration_test`Envoy::Network::ConnectionImpl::onFileEvent(this=0x0000000152076200, events=1) at connection_impl.cc:701:5
frame #54: 0x0000000103aee938 eds_integration_test`Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket>>&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket>>&&, Envoy::StreamInfo::StreamInfo&, bool)::$_7::operator()(this=0x0000600002180eb8, events=1) const at connection_impl.cc:109:9
frame #60: 0x0000000103a9d85c eds_integration_test`std::__1::function<absl::lts_20250512::Status (unsigned int)>::operator()(this=0x0000600002180eb0, __arg=1) const at function.h:989:10
frame #61: 0x0000000103a9d824 eds_integration_test`Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<absl::lts_20250512::Status (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_1::operator()(this=0x0000600002180ea8, events=1) const at dispatcher_impl.cc:202:16
frame #67: 0x0000000103a9d85c eds_integration_test`std::__1::function<absl::lts_20250512::Status (unsigned int)>::operator()(this=0x0000600001a25048, __arg=1) const at function.h:989:10
frame #68: 0x0000000103aa6720 eds_integration_test`Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(this=0x0000600001a24fc0, events=1) at file_event_impl.cc:161:3
frame #69: 0x0000000103aa6ffc eds_integration_test`Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_2::operator()(this=0x00000001700b3d7f, (null)=23, what=34, arg=0x0000600001a24fc0) const at file_event_impl.cc:82:16
frame #70: 0x0000000103aa6d28 eds_integration_test`Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_2::__invoke((null)=23, what=34, arg=0x0000600001a24fc0) at file_event_impl.cc:66:7
frame #71: 0x0000000104a8cce4 eds_integration_test`event_persist_closure(base=0x0000000151f04800, ev=0x0000600001a24fc8) at event.c:1645:9
frame #72: 0x0000000104a8c2e8 eds_integration_test`event_process_active_single_queue(base=0x0000000151f04800, activeq=0x0000600002d20210, max_to_process=2147483647, endtime=0x0000000000000000) at event.c:1704:4
frame #73: 0x0000000104a86798 eds_integration_test`event_process_active(base=0x0000000151f04800) at event.c:1805:9
frame #74: 0x0000000104a855dc eds_integration_test`event_base_loop(base=0x0000000151f04800, flags=0) at event.c:2047:12
frame #75: 0x0000000103ef001c eds_integration_test`Envoy::Event::LibeventScheduler::run(this=0x0000000151f09f60, mode=Block) at libevent_scheduler.cc:61:3