-
Notifications
You must be signed in to change notification settings - Fork 5.3k
QuicHttpIntegrationTest fails about 20% of the time under clang-tsan due to FakeUpstream object lifetime problems #16493
Description
$ bazel test --config=remote --config=clang-tsan --runs_per_test=10 //test/integration:quic_http_integration_test
//test/integration:quic_http_integration_test FAILED in 12 out of 60 in 118.7s
There are several different failures modes, but they boil down to the fake upstream running callbacks in the dispatcher that refer to objects that are already deleted or possibly double deleting objects.
Test cases I have seen fail:
QuicHttpIntegrationTests/QuicHttpIntegrationTest.DownstreamReadDisabledOnGiantPost
QuicHttpIntegrationTests/QuicHttpIntegrationTest.LargeFlowControlOnAndGiantBody
WARNING: ThreadSanitizer: heap-use-after-free (virtual call vs free) (pid=17)
Read of size 8 at 0x7b4800030130 by thread T18:
>0 Envoy::Http::Http1::ConnectionImpl::flushOutput(bool) (quic_http_integration_test+0x62de878)
>1 Envoy::Http::Http1::StreamEncoderImpl::encodeHeadersBase(Envoy::Http::RequestOrResponseHeaderMap const&, std::__1::optional, bool, bool) (quic_http_integration_test+0x62de439)
>2 Envoy::Http::Http1::ResponseEncoderImpl::encodeHeaders(Envoy::Http::ResponseHeaderMap const&, bool) (quic_http_integration_test+0x62e1cf7)
>3 non-virtual thunk to Envoy::Http::Http1::ResponseEncoderImpl::encodeHeaders(Envoy::Http::ResponseHeaderMap const&, bool) (quic_http_integration_test+0x62e1dc2)
>4 Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1::operator()() const (quic_http_integration_test+0x3d486f6)
>5 decltype(std::__1::forward<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(fp)()) std::__1::__invoke<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&) (quic_http_integration_test+0x3d48590)
>6 void std::__1::__invoke_void_return_wrapper::__call<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&) (quic_http_integration_test+0x3d484f0)
>7 std::__1::__function::__alloc_func<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1, std::__1::allocator<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1>, void ()>::operator()() (quic_http_integration_test+0x3d48490)
>8 std::__1::__function::__func<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1, std::__1::allocator<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1>, void ()>::operator()() (quic_http_integration_test+0x3d46b4f)
>9 std::__1::__function::__value_func<void ()>::operator()() const (quic_http_integration_test+0x3c04b86)
>10 std::__1::function<void ()>::operator()() const (quic_http_integration_test+0x3c04618)
>11 Envoy::Event::DispatcherImpl::runPostCallbacks() (quic_http_integration_test+0x78a908c)
.... many more stack frames...
>26 Envoy::FakeUpstream::threadRoutine() (quic_http_integration_test+0x3d3e1d4)
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=17)
Write of size 8 at 0x7b4800030130 by thread T23:
>0 Envoy::Network::ConnectionImpl::~ConnectionImpl() (quic_http_integration_test+0x79521cc)
>1 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797e3df)
>2 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d416)
>3 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d48f)
>4 virtual thunk to Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d71a)
>5 std::__1::default_deleteEnvoy::Network::Connection::operator()(Envoy::Network::Connection*) const (quic_http_integration_test+0x6a31b76)
>6 std::__1::unique_ptr<Envoy::Network::Connection, std::__1::default_deleteEnvoy::Network::Connection >::reset(Envoy::Network::Connection*) (quic_http_integration_test+0x6a31a90)
>7 std::__1::unique_ptr<Envoy::Network::Connection, std::__1::default_deleteEnvoy::Network::Connection >::~unique_ptr() (quic_http_integration_test+0x6a23fac)
>8 Envoy::Server::ActiveTcpConnection::~ActiveTcpConnection() (quic_http_integration_test+0x6a19fd1)
>9 Envoy::Server::ActiveTcpConnection::~ActiveTcpConnection() (quic_http_integration_test+0x6a1a0ef)
>10 std::__1::default_deleteEnvoy::Event::DeferredDeletable::operator()(Envoy::Event::DeferredDeletable*) const (quic_http_integration_test+0x4ba2686)
>11 std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_deleteEnvoy::Event::DeferredDeletable >::reset(Envoy::Event::DeferredDeletable*) (quic_http_integration_test+0x4ba2620)
>12 Envoy::Event::DispatcherImpl::clearDeferredDeleteList() (quic_http_integration_test+0x78a38d5)
... lots of stack frames ...
>27 Envoy::FakeUpstream::threadRoutine() (quic_http_integration_test+0x3d3e1d4)