use-after-free in Init::Manager #6116
Description
We've discovered a reproducible use-after-free in InitManagerImpl. The general pattern is:
- A
Targetregisters itself with anInitManagerImpl. InitManagerImpl::initialize()is called, which invokesTarget::initialize()on the target. It passes a callback argument, a lambda that captures (but does not own)InitManagerImpl. The target, in this case, stores the callback and begins initializing asynchronously.- The
InitManagerImplis freed from the heap. - Finally, the
Targets finishes initializing and invokes its callback, which attempts to use the freedInitManagerImplpreviously captured.
The specific case where we are hitting this is essentially a race between LDS and RDS. I'll try to describe this at both the xDS level, and the code level (in terms of the steps above).
- Start from a state where
ListenerManagerImplhas all its workers started, so anyListenerImplcreated will use its owndynamic_init_manager_rather than the global one (seeListenerImpl::initManager). - LDS receives a listener, which references some route configuration that hasn't been defined yet, so RDS requests that route configuration.
- A
ListenerImplis created for the listener, which owns anInitManagerImpl. - A
RdsRouteConfigSubscriptionis created, and registers itself as a target with the aboveInitManagerImpl(step 1 above). - The
ListenerImplis initialized, which calls initialize on theRdsRouteConfigSubscription. The callback is stored while we wait for the RDS response (step 2 above).
- A
- LDS then receives a new version of the same listener, overwriting the old version.
- The original
ListenerImplis destroyed, which destroys theInitManagerImplit owns (step 3 above).
- The original
- Finally, RDS receives the response for the route configuration it had previously requested.
- The
RdsRouteConfigSubscriptioninvokes its callback, referencing the now deletedInitManagerImpl(step 4 above).
- The
The relevant bits from the actual stack trace:
#0: Envoy::SignalAction::sigHandler() [0x1b6fd0c] source/exe/signal_action.cc:17
#1: __restore_rt [0x7fa1b0d540c0] ??:0
#2: std::__cxx11::list<>::remove() [0x126d1a4] /usr/lib/gcc/x86_64-linux-gnu/8.0.1/../../../../include/c++/8.0.1/bits/list.tcc:335
#3: Envoy::Server::InitManagerImpl::initializeTarget()::$_0::operator()() [0x126c61b] source/server/init_manager_impl.cc:45
#4: std::_Function_handler<>::_M_invoke() [0x126c29d] /usr/lib/gcc/x86_64-linux-gnu/8.0.1/../../../../include/c++/8.0.1/bits/std_function.h:299
#5: std::function<>::operator()() [0x4f645e] /usr/lib/gcc/x86_64-linux-gnu/8.0.1/../../../../include/c++/8.0.1/bits/std_function.h:687
#6: Envoy::Router::RdsRouteConfigSubscription::runInitializeCallbackIfAny() [0xa4ded0] source/common/router/rds_impl.cc:138
#7: Envoy::Router::RdsRouteConfigSubscription::onConfigUpdate() [0xa4e48b] source/common/router/rds_impl.cc:?
#8: Envoy::Config::GrpcMuxSubscriptionImpl<>::onConfigUpdate() [0xa59929] bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:55
#9: Envoy::Config::GrpcMuxImpl::handleResponse() [0x11697f4] source/common/config/grpc_mux_impl.cc:171
#10: Envoy::Config::GrpcStream<>::onReceiveMessage() [0x116b3de] bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:82
#11: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0x116b2a9] bazel-out/k8-dbg/bin/include/envoy/grpc/_virtual_includes/async_client_interface/envoy/grpc/async_client.h:172
#12: Envoy::Grpc::AsyncStreamImpl::onData() [0xd96c8d] source/common/grpc/async_client_impl.cc:142
#13: Envoy::Http::AsyncStreamImpl::encodeData() [0xd9bd35] source/common/http/async_client_impl.cc:105
#14: Envoy::Router::Filter::onUpstreamData() [0x119a13b] source/common/router/router.cc:770
#15: Envoy::Router::Filter::UpstreamRequest::decodeData() [0x119bc67] source/common/router/router.cc:985
#16: Envoy::Http::StreamDecoderWrapper::decodeData() [0x8c65ec] bazel-out/k8-dbg/bin/source/common/http/_virtual_includes/codec_wrappers_lib/common/http/codec_wrappers.h:37
#17: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x12a305b] source/common/http/http2/codec_impl.cc:503
#18: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_9::operator()() [0x12a7aa8] source/common/http/http2/codec_impl.cc:808
#19: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_9::__invoke() [0x12a7a75] source/common/http/http2/codec_impl.cc:807
Decoding that a bit:
- Frames 8 through 6: RDS receives update...
- Frames 5 through 3: RDS calls its stored callback...
- Frame 2: The callback attempts to remove the RdsRouteConfigSubscription from its list of init targets...
- Frames 1 and 0: Oh noes, that list of init targets was previously freed.