Skip to content

QuicHttpIntegrationTest fails about 20% of the time under clang-tsan due to FakeUpstream object lifetime problems #16493

@antoniovicente

Description

@antoniovicente

$ bazel test --config=remote --config=clang-tsan --runs_per_test=10 //test/integration:quic_http_integration_test
//test/integration:quic_http_integration_test FAILED in 12 out of 60 in 118.7s

There are several different failures modes, but they boil down to the fake upstream running callbacks in the dispatcher that refer to objects that are already deleted or possibly double deleting objects.

Test cases I have seen fail:
QuicHttpIntegrationTests/QuicHttpIntegrationTest.DownstreamReadDisabledOnGiantPost
QuicHttpIntegrationTests/QuicHttpIntegrationTest.LargeFlowControlOnAndGiantBody

WARNING: ThreadSanitizer: heap-use-after-free (virtual call vs free) (pid=17)
Read of size 8 at 0x7b4800030130 by thread T18:
>0 Envoy::Http::Http1::ConnectionImpl::flushOutput(bool) (quic_http_integration_test+0x62de878)
>1 Envoy::Http::Http1::StreamEncoderImpl::encodeHeadersBase(Envoy::Http::RequestOrResponseHeaderMap const&, std::__1::optional, bool, bool) (quic_http_integration_test+0x62de439)
>2 Envoy::Http::Http1::ResponseEncoderImpl::encodeHeaders(Envoy::Http::ResponseHeaderMap const&, bool) (quic_http_integration_test+0x62e1cf7)
>3 non-virtual thunk to Envoy::Http::Http1::ResponseEncoderImpl::encodeHeaders(Envoy::Http::ResponseHeaderMap const&, bool) (quic_http_integration_test+0x62e1dc2)
>4 Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1::operator()() const (quic_http_integration_test+0x3d486f6)
>5 decltype(std::__1::forward<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(fp)()) std::__1::__invoke<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&) (quic_http_integration_test+0x3d48590)
>6 void std::__1::__invoke_void_return_wrapper::__call<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&>(Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1&) (quic_http_integration_test+0x3d484f0)
>7 std::__1::__function::__alloc_func<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1, std::__1::allocator<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1>, void ()>::operator()() (quic_http_integration_test+0x3d48490)
>8 std::__1::__function::__func<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1, std::__1::allocator<Envoy::FakeStream::encodeHeaders(Envoy::Http::HeaderMap const&, bool)::$_1>, void ()>::operator()() (quic_http_integration_test+0x3d46b4f)
>9 std::__1::__function::__value_func<void ()>::operator()() const (quic_http_integration_test+0x3c04b86)
>10 std::__1::function<void ()>::operator()() const (quic_http_integration_test+0x3c04618)
>11 Envoy::Event::DispatcherImpl::runPostCallbacks() (quic_http_integration_test+0x78a908c)
.... many more stack frames...
>26 Envoy::FakeUpstream::threadRoutine() (quic_http_integration_test+0x3d3e1d4)

WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=17)
Write of size 8 at 0x7b4800030130 by thread T23:
>0 Envoy::Network::ConnectionImpl::~ConnectionImpl() (quic_http_integration_test+0x79521cc)
>1 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797e3df)
>2 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d416)
>3 Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d48f)
>4 virtual thunk to Envoy::Network::ServerConnectionImpl::~ServerConnectionImpl() (quic_http_integration_test+0x797d71a)
>5 std::__1::default_deleteEnvoy::Network::Connection::operator()(Envoy::Network::Connection*) const (quic_http_integration_test+0x6a31b76)
>6 std::__1::unique_ptr<Envoy::Network::Connection, std::__1::default_deleteEnvoy::Network::Connection >::reset(Envoy::Network::Connection*) (quic_http_integration_test+0x6a31a90)
>7 std::__1::unique_ptr<Envoy::Network::Connection, std::__1::default_deleteEnvoy::Network::Connection >::~unique_ptr() (quic_http_integration_test+0x6a23fac)
>8 Envoy::Server::ActiveTcpConnection::~ActiveTcpConnection() (quic_http_integration_test+0x6a19fd1)
>9 Envoy::Server::ActiveTcpConnection::~ActiveTcpConnection() (quic_http_integration_test+0x6a1a0ef)
>10 std::__1::default_deleteEnvoy::Event::DeferredDeletable::operator()(Envoy::Event::DeferredDeletable*) const (quic_http_integration_test+0x4ba2686)
>11 std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_deleteEnvoy::Event::DeferredDeletable >::reset(Envoy::Event::DeferredDeletable*) (quic_http_integration_test+0x4ba2620)
>12 Envoy::Event::DispatcherImpl::clearDeferredDeleteList() (quic_http_integration_test+0x78a38d5)
... lots of stack frames ...
>27 Envoy::FakeUpstream::threadRoutine() (quic_http_integration_test+0x3d3e1d4)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions