Segfault parsing xDS response with new v3 field and v2 typeUrl #13681
Description
Title: Segfault parsing v3 xDS response to v2 request
Description:
We’re in the process of migrating to the v3 xDS API. Our control plane is currently sending v3 protos to both v2 and v3 endpoints after appropriately changing the typeUrl.
We recently tried setting in our v3 CDS responses a proto field that’s only present on the v3 API (config.cluster.v3.Cluster.LeastRequestLbConfig.active_requests_bias) and observed that sidecars using v2 resource/transport versions segfaulted upon on receiving a proto with a new v3 field and v2 typeUrl.
The crash happens on the VersionConverter::annotateWithOriginalType() method. See the attached backtrace for more details.
As far as I can tell sending responses with new v3 fields and a v2 typeUrl fields is not supported. However it would be much nicer to reject the updates and log an error + increment a stat instead of crashing.
Call Stack:
Backtrace:
#0 0x0000561a05943854 in google::protobuf::Descriptor::full_name (this=0x0) at external/com_google_protobuf/src/google/protobuf/descriptor.h:1889
#0 0x0000561a05943854 in google::protobuf::Descriptor::full_name (this=0x0) at external/com_google_protobuf/src/google/protobuf/descriptor.h:1889
#1 Envoy::Config::VersionConverter::annotateWithOriginalType(google::protobuf::Descriptor const&, google::protobuf::Message&)::TypeAnnotatingProtoVisitor::onMessage(google::protobuf::Message&, void const*) (this=<optimized out>, message=..., ctxt=0x0) at external/envoy/source/common/config/version_converter.cc:84
#2 0x0000561a05943b91 in Envoy::ProtobufMessage::traverseMutableMessage (visitor=..., message=..., ctxt=0x0) at external/envoy/source/common/protobuf/visitor.cc:7
#3 0x0000561a05943d4f in Envoy::ProtobufMessage::traverseMutableMessage (visitor=..., message=..., ctxt=0x561a08a7e240) at external/envoy/source/common/protobuf/visitor.cc:22
#4 0x0000561a05943d4f in Envoy::ProtobufMessage::traverseMutableMessage (visitor=..., message=..., ctxt=0x561a08c08240) at external/envoy/source/common/protobuf/visitor.cc:22
#5 0x0000561a059430df in Envoy::Config::VersionConverter::annotateWithOriginalType (prev_descriptor=..., upgraded_message=...) at external/envoy/source/common/config/version_converter.cc:115
#6 0x0000561a0593c057 in Envoy::MessageUtil::unpackTo (any_message=..., message=...) at external/envoy/source/common/protobuf/utility.cc:591
#7 0x0000561a055c2997 in Envoy::MessageUtil::anyConvert<envoy::config::cluster::v3::Cluster> (message=..., typed_message=...) at bazel-out/k8-opt/bin/external/envoy/source/common/protobuf/_virtual_includes/utility_lib/common/protobuf/utility.h:307
#8 Envoy::MessageUtil::anyConvertAndValidate<envoy::config::cluster::v3::Cluster> (message=..., typed_message=..., validation_visitor=...) at bazel-out/k8-opt/bin/external/envoy/source/common/protobuf/_virtual_includes/utility_lib/common/protobuf/utility.h:328
#9 Envoy::Config::OpaqueResourceDecoderImpl<envoy::config::cluster::v3::Cluster>::decodeResource (this=<optimized out>, resource=...) at bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/opaque_resource_decoder_lib/common/config/opaque_resource_decoder_impl.h:22
#10 0x0000561a0561301f in Envoy::Config::DecodedResourceImpl::DecodedResourceImpl (this=0x561a0916ca80, resource_decoder=..., name=..., aliases=..., resource=..., has_resource=true, version=...) at bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/decoded_resource_lib/common/config/decoded_resource_impl.h:48
#11 0x0000561a0561170a in Envoy::Config::DecodedResourceImpl::DecodedResourceImpl (this=0x561a060bead8 <descriptor_table_envoy_2fconfig_2fcore_2fv3_2fbase_2eproto_once>, resource_decoder=..., resource=..., version=...) at bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/decoded_resource_lib/common/config/decoded_resource_impl.h:25
#12 0x0000561a0562623c in Envoy::Config::GrpcMuxImpl::onDiscoveryResponse (this=0x561a09124c98, message=..., control_plane_stats=...) at external/envoy/source/common/config/grpc_mux_impl.cc:198
#13 0x0000561a055b4b2f in Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DiscoveryResponse>::onReceiveMessageRaw (this=0x561a09124ca8, response=...) at bazel-out/k8-opt/bin/external/envoy/source/common/grpc/_virtual_includes/typed_async_client_lib/common/grpc/typed_async_client.h:151
#14 0x0000561a05664c77 in Envoy::Grpc::AsyncStreamImpl::onData (this=0x561a090ab560, data=..., end_stream=false) at external/envoy/source/common/grpc/async_client_impl.cc:144
#15 0x0000561a0566afb8 in Envoy::Http::AsyncStreamImpl::encodeData (this=0x561a0907c400, data=..., end_stream=<optimized out>) at external/envoy/source/common/http/async_client_impl.cc:117
#16 0x0000561a05681bfe in Envoy::Router::UpstreamRequest::decodeData (this=0x561a09158e00, data=..., end_stream=false) at external/envoy/source/common/router/upstream_request.cc:148
#17 0x0000561a055e5feb in Envoy::Http::ResponseDecoderWrapper::decodeData (this=0x561a091f7d20, data=..., end_stream=false) at bazel-out/k8-opt/bin/external/envoy/source/common/http/_virtual_includes/codec_wrappers_lib/common/http/codec_wrappers.h:35
#18 0x0000561a05712218 in Envoy::Http::Http2::ConnectionImpl::onFrameReceived (this=0x561a090ab210, frame=0x561a08c469a0) at external/envoy/source/common/http/http2/codec_impl.cc:702
#19 0x0000561a0590b3ac in session_call_on_frame_received (session=0x561a08c46800, frame=0x561a08c469a0) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/41/execroot/envoy_lyft/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:3299
#20 nghttp2_session_on_data_received (session=0x561a08c46800, frame=0x561a08c469a0) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/41/execroot/envoy_lyft/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:4975
#21 0x0000561a0590d4c4 in session_process_data_frame (session=0x561a08c46800) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/41/execroot/envoy_lyft/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:4994
#22 nghttp2_session_mem_recv (session=<optimized out>, in=<optimized out>, inlen=<optimized out>) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/41/execroot/envoy_lyft/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:6614
#23 0x0000561a057117a9 in Envoy::Http::Http2::ConnectionImpl::innerDispatch (this=0x561a090ab210, data=...) at external/envoy/source/common/http/http2/codec_impl.cc:529
#24 0x0000561a05719383 in Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6::operator()(Envoy::Buffer::Instance&) const (this=<optimized out>, data=...) at external/envoy/source/common/http/http2/codec_impl.cc:517
#25 std::__1::__invoke<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6&, Envoy::Buffer::Instance&> (__f=..., __args=...) at /opt/llvm/bin/../include/c++/v1/type_traits:3530
#26 std::__1::__invoke_void_return_wrapper<absl::Status>::__call<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6&, Envoy::Buffer::Instance&>(Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6&, Envoy::Buffer::Instance&) (__args=..., __args=...) at /opt/llvm/bin/../include/c++/v1/__functional_base:317
#27 std::__1::__function::__alloc_func<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6, std::__1::allocator<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6>, absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) (this=<optimized out>, __arg=...) at /opt/llvm/bin/../include/c++/v1/functional:1533
#28 std::__1::__function::__func<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6, std::__1::allocator<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_6>, absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) (this=<optimized out>, __arg=...) at /opt/llvm/bin/../include/c++/v1/functional:1707
#29 0x0000561a05791857 in std::__1::__function::__value_func<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const (this=<optimized out>, __args=...) at /opt/llvm/bin/../include/c++/v1/functional:1860
#30 std::__1::function<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const (this=<optimized out>, __arg=...) at /opt/llvm/bin/../include/c++/v1/functional:2419
#31 Envoy::Http::Utility::exceptionToStatus(std::__1::function<absl::Status (Envoy::Buffer::Instance&)>, Envoy::Buffer::Instance&) (dispatch=..., data=...) at external/envoy/source/common/http/utility.cc:40
#32 0x0000561a057115b1 in virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) () at external/envoy/source/common/http/http2/codec_impl.cc:516
#33 0x0000561a0565a860 in Envoy::Http::CodecClient::onData (this=0x561a091c8870, data=...) at external/envoy/source/common/http/codec_client.cc:125
#34 0x0000561a0565b6bd in Envoy::Http::CodecClient::CodecReadFilter::onData (this=<optimized out>, data=...) at bazel-out/k8-opt/bin/external/envoy/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:176
#35 0x0000561a05557133 in Envoy::Network::FilterManagerImpl::onContinueReading (this=0x561a091e8078, filter=<optimized out>, buffer_source=...) at external/envoy/source/common/network/filter_manager_impl.cc:66
#36 0x0000561a0555329e in Envoy::Network::ConnectionImpl::onRead (this=0x561a091e8000, read_buffer_size=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:297
#37 Envoy::Network::ConnectionImpl::onReadReady (this=0x561a091e8000) at external/envoy/source/common/network/connection_impl.cc:579
#38 0x0000561a0555238d in Envoy::Network::ConnectionImpl::onFileEvent (this=0x561a091e8000, events=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:539
#39 0x0000561a0554c606 in std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const (this=0x561a08c02d80, __args=@0x7ffd4250961c: 3) at /opt/llvm/bin/../include/c++/v1/functional:1860
#40 std::__1::function<void (unsigned int)>::operator()(unsigned int) const (this=0x561a08c02d80, __arg=3) at /opt/llvm/bin/../include/c++/v1/functional:2419
#41 Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb (this=0x561a08c02d00, events=3) at external/envoy/source/common/event/file_event_impl.cc:122
#42 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::operator()(int, short, void*) const (this=<optimized out>, what=<optimized out>, arg=0x561a08c02d00) at external/envoy/source/common/event/file_event_impl.cc:95
#43 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*) (what=<optimized out>, arg=0x561a08c02d00) at external/envoy/source/common/event/file_event_impl.cc:79
#44 0x0000561a05900c0b in event_persist_closure (base=0x561a08c8c2c0, ev=<optimized out>) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/1883/execroot/envoy_lyft/external/com_github_libevent_libevent/event.c:1639
#45 event_process_active_single_queue (base=0x561a08c8c2c0, activeq=0x561a08aea6d0, max_to_process=2147483647, endtime=0x0) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/1883/execroot/envoy_lyft/external/com_github_libevent_libevent/event.c:1698
#46 0x0000561a058ff48e in event_process_active (base=0x561a08c8c2c0) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/1883/execroot/envoy_lyft/external/com_github_libevent_libevent/event.c:1802
#47 event_base_loop (base=0x561a08c8c2c0, flags=<optimized out>) at /root/.cache/bazel/_bazel_root/e500e327e6981ab742dbacbb6723d21d/sandbox/processwrapper-sandbox/1883/execroot/envoy_lyft/external/com_github_libevent_libevent/event.c:2041
#48 0x0000561a054cb38f in Envoy::Server::InstanceImpl::run (this=0x561a08c16000) at external/envoy/source/server/server.cc:668
#49 0x0000561a05139668 in Envoy::MainCommonBase::run (this=0x561a08bce2b8) at external/envoy/source/exe/main_common.cc:162
#50 0x0000561a0513a095 in Envoy::MainCommon::run (this=0x561a08bce000) at bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:101
#51 Envoy::MainCommon::main(int, char**, std::__1::function<void (Envoy::Server::Instance&)>) (argc=15, argv=0x7ffd42509ab8, hook=...) at external/envoy/source/exe/main_common.cc:234
#52 0x0000561a05137d1c in main (argc=101444312, argv=0x561a091fdf50) at external/envoy/source/exe/main.cc:12