Update Envoy SHA to latest. by jplevyak · Pull Request #2214 · istio/proxy

jplevyak · 2019-05-14T23:19:50Z

No description provided.

PiotrSikora · 2019-05-15T00:49:50Z

The sanitizer failures look real, you might need to update protobuf version to match upstream.

repositories.bzl

WORKSPACE

jplevyak · 2019-05-15T18:03:55Z

The tsan errors look real and I am using the proto libraries from the new envoy... ideas?

duderino · 2019-05-15T18:16:11Z

Yeah, tsan errors look real:

I0515 17:34:52.624] WARNING: ThreadSanitizer: data race (pid=15)
I0515 17:34:52.624]   Write of size 1 at 0x0000060d49c8 by thread T7 (mutexes: write M237137221860516176):
I0515 17:34:52.625]     #0 event_del_nolock_ <null> (http_filter_integration_test+0x535c50d)
I0515 17:34:52.625]     #1 event_process_active_single_queue <null> (http_filter_integration_test+0x53661fe)
I0515 17:34:52.625]     #2 event_base_loop <null> (http_filter_integration_test+0x53603ab)
I0515 17:34:52.625]     #3 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) <null> (http_filter_integration_test+0x482b745)
I0515 17:34:52.625]     #4 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) <null> (http_filter_integration_test+0x45196c8)
I0515 17:34:52.626]     #5 Envoy::Server::GuardDogImpl::start(Envoy::Api::Api&)::$_3::operator()() const <null> (http_filter_integration_test+0x2e98008)
I0515 17:34:52.626]     #6 std::_Function_handler<void (), Envoy::Server::GuardDogImpl::start(Envoy::Api::Api&)::$_3>::_M_invoke(std::_Any_data const&) <null> (http_filter_integration_test+0x2e97d8a)
I0515 17:34:52.626]     #7 std::function<void ()>::operator()() const <null> (http_filter_integration_test+0x1ef6fce)
I0515 17:34:52.626]     #8 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::operator()(void*) const <null> (http_filter_integration_test+0x5335f38)
I0515 17:34:52.626]     #9 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::__invoke(void*) <null> (http_filter_integration_test+0x5335ec8)
I0515 17:34:52.627] 
I0515 17:34:52.627]   Previous write of size 1 at 0x0000060d49c8 by thread T5 (mutexes: write M87392534250335536):
I0515 17:34:52.627]     #0 event_add_nolock_ <null> (http_filter_integration_test+0x535db7e)
I0515 17:34:52.627]     #1 event_add <null> (http_filter_integration_test+0x5363556)
I0515 17:34:52.627]     #2 Envoy::Event::SignalEventImpl::SignalEventImpl(Envoy::Event::DispatcherImpl&, int, std::function<void ()>) <null> (http_filter_integration_test+0x45231ec)
I0515 17:34:52.627]     #3 Envoy::Event::DispatcherImpl::listenForSignal(int, std::function<void ()>) <null> (http_filter_integration_test+0x451937f)
I0515 17:34:52.628]     #4 Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::function<void ()>) <null> (http_filter_integration_test+0x2e595cb)
I0515 17:34:52.628]     #5 Envoy::Server::InstanceImpl::run() <null> (http_filter_integration_test+0x2e59fba)
I0515 17:34:52.628]     #6 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::unique_ptr<Envoy::Runtime::RandomGenerator, std::default_delete<Envoy::Runtime::RandomGenerator> >&&) <null> (http_filter_integration_test+0x2d9f7b6)
I0515 17:34:52.629]     #7 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool) <null> (http_filter_integration_test+0x2d9f35c)
I0515 17:34:52.629]     #8 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::function<void ()>, bool, bool)::$_0::operator()() const <null> (http_filter_integration_test+0x2da09be)
I0515 17:34:52.629]     #9 std::_Function_handler<void (), Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::function<void ()>, bool, bool)::$_0>::_M_invoke(std::_Any_data const&) <null> (http_filter_integration_test+0x2da072a)
I0515 17:34:52.629]     #10 std::function<void ()>::operator()() const <null> (http_filter_integration_test+0x1ef6fce)
I0515 17:34:52.629]     #11 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::operator()(void*) const <null> (http_filter_integration_test+0x5335f38)
I0515 17:34:52.629]     #12 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::__invoke(void*) <null> (http_filter_integration_test+0x5335ec8)
I0515 17:34:52.629] 
I0515 17:34:52.630]   Location is global 'event_debug_mode_too_late' of size 1 at 0x0000060d49c8 (http_filter_integration_test+0x0000060d49c8)
I0515 17:34:52.630] 
I0515 17:34:52.630]   Mutex M237137221860516176 is already destroyed.
I0515 17:34:52.630] 
I0515 17:34:52.630]   Mutex M87392534250335536 is already destroyed.
I0515 17:34:52.630] 
I0515 17:34:52.630]   Thread T7 (tid=24, running) created by thread T5 at:
I0515 17:34:52.630]     #0 pthread_create <null> (http_filter_integration_test+0x19b84d5)
I0515 17:34:52.630]     #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>) <null> (http_filter_integration_test+0x53356d4)
I0515 17:34:52.630]     #2 std::_MakeUniq<Envoy::Thread::ThreadImplPosix>::__single_object std::make_unique<Envoy::Thread::ThreadImplPosix, std::function<void ()>&>(std::function<void ()>&) <null> (http_filter_integration_test+0x5336275)
I0515 17:34:52.631]     #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::function<void ()>) <null> (http_filter_integration_test+0x5335b3b)
I0515 17:34:52.631]     #4 Envoy::Server::GuardDogImpl::start(Envoy::Api::Api&) <null> (http_filter_integration_test+0x2e955b9)
I0515 17:34:52.631]     #5 Envoy::Server::GuardDogImpl::GuardDogImpl(Envoy::Stats::Scope&, Envoy::Server::Configuration::Main const&, Envoy::Api::Api&, std::unique_ptr<Envoy::Server::GuardDogImpl::TestInterlockHook, std::default_delete<Envoy::Server::GuardDogImpl::TestInterlockHook> >&&) <null> (http_filter_integration_test+0x2e95072)
I0515 17:34:52.631]     #6 Envoy::Server::GuardDogImpl::GuardDogImpl(Envoy::Stats::Scope&, Envoy::Server::Configuration::Main const&, Envoy::Api::Api&) <null> (http_filter_integration_test+0x2e95790)
I0515 17:34:52.631]     #7 std::_MakeUniq<Envoy::Server::GuardDogImpl>::__single_object std::make_unique<Envoy::Server::GuardDogImpl, Envoy::Stats::StoreRoot&, Envoy::Server::Configuration::MainImpl&, Envoy::Api::Api&>(Envoy::Stats::StoreRoot&, Envoy::Server::Configuration::MainImpl&, Envoy::Api::Api&) <null> (http_filter_integration_test+0x2e6820b)
I0515 17:34:52.632]     #8 Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options const&, std::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&, Envoy::ListenerHooks&) <null> (http_filter_integration_test+0x2e56c31)
I0515 17:34:52.632]     #9 Envoy::Server::InstanceImpl::InstanceImpl(Envoy::Server::Options const&, Envoy::Event::TimeSystem&, std::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Server::HotRestart&, Envoy::Stats::StoreRoot&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::unique_ptr<Envoy::Runtime::RandomGenerator, std::default_delete<Envoy::Runtime::RandomGenerator> >&&, Envoy::ThreadLocal::Instance&, Envoy::Thread::ThreadFactory&, Envoy::Filesystem::Instance&) <null> (http_filter_integration_test+0x2e518f2)
I0515 17:34:52.632]     #10 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::unique_ptr<Envoy::Runtime::RandomGenerator, std::default_delete<Envoy::Runtime::RandomGenerator> >&&) <null> (http_filter_integration_test+0x2d9f6b2)
I0515 17:34:52.632]     #11 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool) <null> (http_filter_integration_test+0x2d9f35c)
I0515 17:34:52.633]     #12 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::function<void ()>, bool, bool)::$_0::operator()() const <null> (http_filter_integration_test+0x2da09be)
I0515 17:34:52.633]     #13 std::_Function_handler<void (), Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::function<void ()>, bool, bool)::$_0>::_M_invoke(std::_Any_data const&) <null> (http_filter_integration_test+0x2da072a)
I0515 17:34:52.633]     #14 std::function<void ()>::operator()() const <null> (http_filter_integration_test+0x1ef6fce)
I0515 17:34:52.633]     #15 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::operator()(void*) const <null> (http_filter_integration_test+0x5335f38)
I0515 17:34:52.633]     #16 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>)::$_0::__invoke(void*) <null> (http_filter_integration_test+0x5335ec8)
I0515 17:34:52.633] 
I0515 17:34:52.633]   Thread T5 (tid=22, running) created by main thread at:
I0515 17:34:52.634]     #0 pthread_create <null> (http_filter_integration_test+0x19b84d5)
I0515 17:34:52.634]     #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>) <null> (http_filter_integration_test+0x53356d4)
I0515 17:34:52.634]     #2 std::_MakeUniq<Envoy::Thread::ThreadImplPosix>::__single_object std::make_unique<Envoy::Thread::ThreadImplPosix, std::function<void ()>&>(std::function<void ()>&) <null> (http_filter_integration_test+0x5336275)
I0515 17:34:52.634]     #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::function<void ()>) <null> (http_filter_integration_test+0x5335b3b)
I0515 17:34:52.634]     #4 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::function<void ()>, bool, bool) <null> (http_filter_integration_test+0x2d9deda)
I0515 17:34:52.634]     #5 Envoy::IntegrationTestServer::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Envoy::Network::Address::IpVersion, std::function<void ()>, bool, Envoy::Event::TestTimeSystem&, Envoy::Api::Api&, bool) <null> (http_filter_integration_test+0x2d9dac6)
I0515 17:34:52.635]     #6 Envoy::BaseIntegrationTest::createGeneratedApiTestServer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) <null> (http_filter_integration_test+0x2d2d317)
I0515 17:34:52.635]     #7 Envoy::BaseIntegrationTest::createEnvoy() <null> (http_filter_integration_test+0x2d2d11d)
I0515 17:34:52.635]     #8 Envoy::BaseIntegrationTest::initialize() <null> (http_filter_integration_test+0x2d2bee4)
I0515 17:34:52.635]     #9 Envoy::(anonymous namespace)::AuthenticationFilterIntegrationTest_EmptyPolicy_Test::TestBody() <null> (http_filter_integration_test+0x1a41c62)
I0515 17:34:52.635]     #10 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (http_filter_integration_test+0x55180b6)
I0515 17:34:52.635]     #11 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (http_filter_integration_test+0x54fe911)
I0515 17:34:52.636]     #12 testing::Test::Run() <null> (http_filter_integration_test+0x54df41b)
I0515 17:34:52.636]     #13 testing::TestInfo::Run() <null> (http_filter_integration_test+0x54e0559)
I0515 17:34:52.636]     #14 testing::TestSuite::Run() <null> (http_filter_integration_test+0x54e0fc4)
I0515 17:34:52.636]     #15 testing::internal::UnitTestImpl::RunAllTests() <null> (http_filter_integration_test+0x54f3e84)
I0515 17:34:52.636]     #16 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) <null> (http_filter_integration_test+0x551d9a6)
I0515 17:34:52.636]     #17 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) <null> (http_filter_integration_test+0x55027a7)
I0515 17:34:52.636]     #18 testing::UnitTest::Run() <null> (http_filter_integration_test+0x54f38ee)
I0515 17:34:52.637]     #19 RUN_ALL_TESTS() <null> (http_filter_integration_test+0x4174ab7)
I0515 17:34:52.637]     #20 Envoy::TestRunner::RunTests(int, char**) <null> (http_filter_integration_test+0x417470d)
I0515 17:34:52.637]     #21 main <null> (http_filter_integration_test+0x4172e79)
I0515 17:34:52.637] 
I0515 17:34:52.637] SUMMARY: ThreadSanitizer: data race (/home/bootstrap/.cache/bazel/_bazel_bootstrap/1487d9a98de60e2921e308f5e5b22a2c/execroot/__main__/bazel-out/k8-fastbuild/bin/src/envoy/http/authn/http_filter_integration_test+0x535c50d) in event_del_nolock_

duderino · 2019-05-15T18:23:18Z

Looks like @htuch has been down this path already: https://www.bountysource.com/issues/70642822-event_debug_mode_too_late-is-thread-unsafe

jplevyak · 2019-05-15T18:32:28Z

From my investigations it seems this is an old issue. One possible fix is the set EVENT__DISABLE_DEBUG_MODE in the libevent build in envoy. The problem is that variable is set outside of locks in libevent.

jplevyak · 2019-05-15T18:34:29Z

If it is a known issue, can we merge without fixing?

duderino · 2019-05-15T18:36:21Z

Maybe. I’d like to understand when/how it was introduced

htuch · 2019-05-15T19:00:34Z

I'm amused this ended up on bountrysource.com. We (@akonradi and myself) have reported a number of TSAN issues to libevent:

We have internal Google fixes for these, but we do not have libevent buy-in for some of these fixes and in addition need tests etc. to upstream. Would be great if someone wants to work on them.

repositories.bzl

duderino · 2019-05-15T21:20:10Z

I'm amused this ended up on bountrysource.com. We (@akonradi and myself) have reported a number of TSAN issues to libevent:

libevent/libevent#779

libevent/libevent#741

libevent/libevent#777

We have internal Google fixes for these, but we do not have libevent buy-in for some of these fixes and in addition need tests etc. to upstream. Would be great if someone wants to work on them.

@htuch what did Envoy do in the meantime? Suppress the warnings?

tsan.suppressions

in libevent.

Makefile

PiotrSikora · 2019-05-15T22:17:48Z

WORKSPACE

 # Determine SHA256 `wget https://github.com/envoyproxy/envoy/archive/COMMIT.tar.gz && sha256sum COMMIT.tar.gz`
-ENVOY_SHA = "5ea1a0c1cb506ed3e80d52b572b0f767f55f9f39"
+# envoy commit date  05/15/2019
+# bazel version: 0.25.0


Nit: this isn't really true, since Bazel version depends on the build image / CI environment, and 0.25.x is only used on macOS right now, since it's pulled at build time.

but it does work for 0.25... I think Yuchen wanted this so he could find a version that would work.

Fair enough, but it uses 0.22 for Linux builds (CircleCI & Prow), 0.25 is only used for macOS.

Let's merge this and I will create the correct a circleci image in which has bazel 0.25

repositories.bzl

PiotrSikora · 2019-05-15T23:13:57Z

/lgtm
/approve

duderino · 2019-05-15T23:55:10Z

we had a discussion with @htuch and learned that envoy too suppresses these specific libevent tsan errors so we're doing the same here.

/lgtm

istio-testing · 2019-05-15T23:55:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: duderino, jplevyak, PiotrSikora

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [duderino]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Update Envoy SHA to latest.

4685fa7

googlebot added the cla: yes Set by the Google CLA bot to indicate the author of a PR has signed the Google CLA. label May 14, 2019

istio-testing requested review from JimmyCYJ and qiwzhang May 14, 2019 23:19

jplevyak requested review from PiotrSikora, duderino and kyessenov and removed request for qiwzhang May 14, 2019 23:20

jplevyak added 3 commits May 14, 2019 16:22

Fix formatting.

fb94613

Downgrade bazel version for macos.

4808ca9

Fix build with bazel 0.25.

367455d

Remove duplicate dependency.

1108a56

PiotrSikora reviewed May 15, 2019

View reviewed changes

repositories.bzl Outdated Show resolved Hide resolved

PiotrSikora reviewed May 15, 2019

View reviewed changes

repositories.bzl Outdated Show resolved Hide resolved

lambdai reviewed May 15, 2019

View reviewed changes

WORKSPACE Show resolved Hide resolved

jplevyak added 2 commits May 15, 2019 10:08

Address comments.

104ae09

Address comments.

dedc44b

Supress tsan issue in libevent.

2e111f4

PiotrSikora reviewed May 15, 2019

View reviewed changes

repositories.bzl Outdated Show resolved Hide resolved

jplevyak added 2 commits May 15, 2019 14:10

Update supression file.

dafc46c

typo

513ba50

lambdai reviewed May 15, 2019

View reviewed changes

tsan.suppressions Show resolved Hide resolved

jplevyak added 3 commits May 15, 2019 14:52

Remove duplicate protobuf library. Add quotes on supressions argument.

1e9b644

Add -DEVENT__DISABLE_DEBUG_MODE for tsan targets to work around race

368c242

in libevent.

Fix typo, remove disabling of debug mode.

9582136

PiotrSikora reviewed May 15, 2019

View reviewed changes

jplevyak added 2 commits May 15, 2019 15:29

Remove quotes.

5db29ee

Remove protobuf.bzl.

1c8506d

PiotrSikora approved these changes May 15, 2019

View reviewed changes

istio-testing assigned PiotrSikora May 15, 2019

istio-testing added the lgtm label May 15, 2019

istio-testing assigned duderino May 15, 2019

istio-testing added the approved label May 15, 2019

istio-testing merged commit 0b76a30 into master May 15, 2019

nrjpoddar mentioned this pull request Jun 5, 2019

Build 1.1.8 part 2 #2259

Merged

Conversation

jplevyak commented May 14, 2019

Uh oh!

PiotrSikora commented May 15, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jplevyak commented May 15, 2019

Uh oh!

duderino commented May 15, 2019

Uh oh!

duderino commented May 15, 2019

Uh oh!

jplevyak commented May 15, 2019

Uh oh!

jplevyak commented May 15, 2019

Uh oh!

duderino commented May 15, 2019

Uh oh!

htuch commented May 15, 2019

Uh oh!

Uh oh!

duderino commented May 15, 2019

Uh oh!

Uh oh!

Uh oh!

PiotrSikora May 15, 2019

Choose a reason for hiding this comment

Uh oh!

jplevyak May 15, 2019

Choose a reason for hiding this comment

Uh oh!

PiotrSikora May 15, 2019

Choose a reason for hiding this comment

Uh oh!

lambdai May 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

PiotrSikora commented May 15, 2019

Uh oh!

duderino commented May 15, 2019

Uh oh!

istio-testing commented May 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants