TSAN error triggered by std::string race in FakeRawConnection::write #944
Description
From debugging prints etc. it seems that memcpy from data.c_str() in the closure in FakeRawConnection::write() is racing with delete on std::string in the main thread execution action closure.
In gcc-4.9, the std::basic_string implementation is using reference counting and CoW. It's unclear exactly why the reference counting is not working correctly across threads, but there are some known issues (e.g. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334).
The workaround is to ensure the underlying std::string that is shared across threads stays alive for the test.
This may just be an artifact of TSAN, or affect other parts of Envoy. This issue should stay open until we understand better what's going on.
WARNING: ThreadSanitizer: data race (pid=7)
Write of size 8 at 0x7b080001fdd8 by main thread:
#0 operator delete(void*) /proc/self/cwd/third_party/llvm/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 (integration_test+0x53b570)
#1 operator() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/faa4a85d-81af-4127-a69d-f89c149adef2-1/execroot/envoy/test/integration/integration_t
est.cc:158 (integration_test+0x54725d)
#2 std::_Function_handler<void (), IntegrationTest_TcpProxyDownstreamDisconnect_Test::TestBody()::$_17>::_M_invoke(std::_Any_data const&) /google/src/files/151379707/depot/google3/third_party/crosstoo
l/v18/llvm_unstable/wrappers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2039 (integration_test+0x546f4a
)
#3 std::function<void ()>::operator()() const /google/src/files/151379707/depot/google3/third_party/crosstool/v18/llvm_unstable/wrappers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.
x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2440 (integration_test+0x55747e)
#4 BaseIntegrationTest::executeActions(std::list<std::function<void ()>, std::allocator<std::function<void ()> > >) /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a445
97f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-138/execroot/envoy/./test/integration/integration.h:157 (integration_test+0x54b535)
#5 IntegrationTest_TcpProxyDownstreamDisconnect_Test::TestBody() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/faa4a85d-81af-4127-a69d-f89c149ad
ef2-1/execroot/envoy/test/integration/integration_test.cc:153 (integration_test+0x53ea44)
#6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/external/envoy_deps_cache_55b3dedbe251305eb00b70c97827640c/googletest.dep.build/googletest-release-1.8.0/googletest/src/gtest.cc:2402 (integration_test+0x102c4c6)
#7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676
793239ac96d94294e6c7a44597f/external/envoy_deps_cache_55b3dedbe251305eb00b70c97827640c/googletest.dep.build/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (integration_test+0x102c4c6)
#8 TestRunner::RunTests(int, char**) /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-107/execroot/envoy/./tes
t/test_runner.h:32 (integration_test+0xe60e79)
#9 main /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-107/execroot/envoy/test/main.cc:32 (integration_test+
0xe5ff8e)
Previous read of size 1 at 0x7b080001fddc by thread T1:
#0 memcpy /proc/self/cwd/third_party/llvm/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:696 (integration_test+0x4e73c0)
#1 evbuffer_add /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/external/envoy_deps_cache_55b3dedbe251305eb00b70c97827640c/libevent.dep.build/libevent-2.1.8-sta
ble/buffer.c:? (integration_test+0xf15f06)
#2 OwnedImpl /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-73/execroot/envoy/source/common/buffer/buffer_im
pl.cc:106 (integration_test+0xf12ab7)
#3 operator() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/faa4a85d-81af-4127-a69d-f89c149adef2-0/execroot/envoy/test/integration/fake_upstream
.cc:294 (integration_test+0x591811)
#4 std::_Function_handler<void (), FakeRawConnection::write(std::string const&)::$_8>::_M_invoke(std::_Any_data const&) /google/src/files/151379707/depot/google3/third_party/crosstool/v18/llvm_unstabl
e/wrappers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2039 (integration_test+0x59144a)
#5 std::function<void ()>::operator()() const /google/src/files/151379707/depot/google3/third_party/crosstool/v18/llvm_unstable/wrappers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.
x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2440 (integration_test+0x55747e)
#6 Event::DispatcherImpl::runPostCallbacks() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-96/execroot/envo
y/source/common/event/dispatcher_impl.cc:152 (integration_test+0x681dd7)
#7 operator() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-96/execroot/envoy/source/common/event/dispatche
r_impl.cc:28 (integration_test+0x6832a8)
#8 std::_Function_handler<void (), Event::DispatcherImpl::DispatcherImpl()::$_1>::_M_invoke(std::_Any_data const&) /google/src/files/151379707/depot/google3/third_party/crosstool/v18/llvm_unstable/wra
ppers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2039 (integration_test+0x682fda)
#9 std::function<void ()>::operator()() const /google/src/files/151379707/depot/google3/third_party/crosstool/v18/llvm_unstable/wrappers/bin/../../toolchain/bin/../lib/gcc/x86_64-grtev4-linux-gnu/4.9.
x-google/../../../../x86_64-grtev4-linux-gnu/include/c++/4.9.x-google/functional:2440 (integration_test+0x55747e)
#10 operator() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-106/execroot/envoy/source/common/event/timer_i
mpl.cc:14 (integration_test+0x68b616)
#11 __invoke /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/0d694ed6-27b8-4532-ae2e-779131fca1cd-106/execroot/envoy/source/common/event/timer_imp
l.cc:14 (integration_test+0x68b597)
#12 event_process_active_single_queue /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/external/envoy_deps_cache_55b3dedbe251305eb00b70c97827640c/libevent.dep.bu
ild/libevent-2.1.8-stable/event.c:1646 (integration_test+0xf24432)
#13 FakeUpstream::threadRoutine() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/faa4a85d-81af-4127-a69d-f89c149adef2-0/execroot/envoy/test/integ
ration/fake_upstream.cc:247 (integration_test+0x58c586)
#14 operator() /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/bazel-sandbox/faa4a85d-81af-4127-a69d-f89c149adef2-0/execroot/envoy/test/integration/fake_upstrea
m.cc:220 (integration_test+0x591118)