-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Closed
Labels
Description
//test/integration/sds_dynamic_integration_test fails intermittently. I've seen this in a couple of PRs, but also on master: https://circleci.com/gh/envoyproxy/envoy/91813
Potentially related is the following assertion in the same test file (from https://circleci.com/gh/envoyproxy/envoy/92168):
==================== Test output for //test/integration:sds_dynamic_integration_test:
WARNING: Perftools heap leak checker is active -- Performance may suffer
[==========] Running 16 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 8 tests from IpVersionsClientType/SdsDynamicDownstreamIntegrationTest
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/0
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/0 (147 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/1
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/1 (115 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/2
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/2 (110 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/3
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.BasicSuccess/3 (117 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/0
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/0 (126 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/1
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/1 (123 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/2
[ OK ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/2 (120 ms)
[ RUN ] IpVersionsClientType/SdsDynamicDownstreamIntegrationTest.WrongSecretFirst/3
E0905 18:22:37.304682627 6216 call.cc:1933] assertion failed: grpc_cq_begin_op(call->cq, notify_tag)
[2018-09-05 18:22:37.304][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0x17fb
[2018-09-05 18:22:37.305][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<6216> obj</lib/x86_64-linux-gnu/libc.so.6> (If unsymbolized, use tools/stack_decode.py):
[2018-09-05 18:22:37.305][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<6216> #0 0x7fce25efb428 (unknown)
[2018-09-05 18:22:37.305][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<6216> #1 0x7fce25efd029 (unknown)
[2018-09-05 18:22:37.305][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<6216> obj</build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/bazel-out/k8-fastbuild/bin/test/integration/sds_dynamic_integration_test.runfiles/ci/test/integration/sds_dynamic_integration_test>
[2018-09-05 18:22:37.312][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #2 0x87cac1 call_start_batch()
[2018-09-05 18:22:37.320][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #3 0x87b80c grpc_call_start_batch
[2018-09-05 18:22:37.327][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #4 0x7afc31 grpc::Channel::PerformOpsOnCall()
[2018-09-05 18:22:37.334][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #5 0x78d69e grpc::internal::Call::PerformOps()
[2018-09-05 18:22:37.341][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #6 0x784947 grpc::ClientAsyncReaderWriter<>::Write()
[2018-09-05 18:22:37.348][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #7 0x77fc54 Envoy::Grpc::GoogleAsyncStreamImpl::writeQueued()
[2018-09-05 18:22:37.356][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #8 0x77f951 Envoy::Grpc::GoogleAsyncStreamImpl::sendMessage()
[2018-09-05 18:22:37.363][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #9 0xfdabe0 Envoy::Config::GrpcMuxImpl::sendDiscoveryRequest()
[2018-09-05 18:22:37.370][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #10 0xfde9d0 Envoy::Config::GrpcMuxImpl::GrpcMuxWatchImpl::~GrpcMuxWatchImpl()
[2018-09-05 18:22:37.377][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #11 0xfdea68 Envoy::Config::GrpcMuxImpl::GrpcMuxWatchImpl::~GrpcMuxWatchImpl()
[2018-09-05 18:22:37.384][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #12 0x6c027e std::default_delete<>::operator()()
[2018-09-05 18:22:37.392][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #13 0x6bfb42 std::unique_ptr<>::~unique_ptr()
[2018-09-05 18:22:37.399][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #14 0xa97adb Envoy::Config::GrpcMuxSubscriptionImpl<>::~GrpcMuxSubscriptionImpl()
[2018-09-05 18:22:37.406][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #15 0xa979a2 Envoy::Config::GrpcSubscriptionImpl<>::~GrpcSubscriptionImpl()
[2018-09-05 18:22:37.413][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #16 0xa979e8 Envoy::Config::GrpcSubscriptionImpl<>::~GrpcSubscriptionImpl()
[2018-09-05 18:22:37.420][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #17 0xa94ede std::default_delete<>::operator()()
[2018-09-05 18:22:37.427][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #18 0xa93992 std::unique_ptr<>::~unique_ptr()
[2018-09-05 18:22:37.435][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #19 0xa947f1 Envoy::Secret::SdsApi::~SdsApi()
[2018-09-05 18:22:37.442][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #20 0xa92343 __gnu_cxx::new_allocator<>::destroy<>()
[2018-09-05 18:22:37.450][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #21 0xa922d7 std::allocator_traits<>::destroy<>()
[2018-09-05 18:22:37.457][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #22 0xa91f1b std::_Sp_counted_ptr_inplace<>::_M_dispose()
[2018-09-05 18:22:37.464][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #23 0x4ffa3b std::_Sp_counted_base<>::_M_release()
[2018-09-05 18:22:37.471][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #24 0x4ff9e9 std::__shared_count<>::~__shared_count()
[2018-09-05 18:22:37.478][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #25 0xa92568 std::__shared_ptr<>::~__shared_ptr()
[2018-09-05 18:22:37.486][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #26 0xa8a9b4 std::shared_ptr<>::~shared_ptr()
[2018-09-05 18:22:37.493][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #27 0x11777e9 Envoy::Ssl::ContextConfigImpl::~ContextConfigImpl()
[2018-09-05 18:22:37.500][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #28 0x117c3a4 Envoy::Ssl::ServerContextConfigImpl::~ServerContextConfigImpl()
[2018-09-05 18:22:37.507][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #29 0x117a4ef Envoy::Ssl::ServerContextConfigImpl::~ServerContextConfigImpl()
[2018-09-05 18:22:37.515][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #30 0x117a528 Envoy::Ssl::ServerContextConfigImpl::~ServerContextConfigImpl()
[2018-09-05 18:22:37.522][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #31 0x61a50e std::default_delete<>::operator()()
[2018-09-05 18:22:37.529][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #32 0x61a072 std::unique_ptr<>::~unique_ptr()
[2018-09-05 18:22:37.536][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #33 0x1173f2b Envoy::Ssl::ServerSslSocketFactory::~ServerSslSocketFactory()
[2018-09-05 18:22:37.544][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #34 0x1173f68 Envoy::Ssl::ServerSslSocketFactory::~ServerSslSocketFactory()
[2018-09-05 18:22:37.551][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #35 0x50832e std::default_delete<>::operator()()
[2018-09-05 18:22:37.558][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #36 0x508292 std::unique_ptr<>::~unique_ptr()
[2018-09-05 18:22:37.565][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #37 0x675be0 Envoy::Server::FilterChainImpl::~FilterChainImpl()
[2018-09-05 18:22:37.572][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #38 0x675db3 __gnu_cxx::new_allocator<>::destroy<>()
[2018-09-05 18:22:37.580][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #39 0x675d47 std::allocator_traits<>::destroy<>()
[2018-09-05 18:22:37.587][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #40 0x67573b std::_Sp_counted_ptr_inplace<>::_M_dispose()
[2018-09-05 18:22:37.587][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #41 0x4ffa3b std::_Sp_counted_base<>::_M_release()
[2018-09-05 18:22:37.587][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #42 0x4ff9e9 std::__shared_count<>::~__shared_count()
[2018-09-05 18:22:37.594][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #43 0x5d7288 std::__shared_ptr<>::~__shared_ptr()
[2018-09-05 18:22:37.601][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #44 0x5d4b64 std::shared_ptr<>::~shared_ptr()
[2018-09-05 18:22:37.608][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #45 0x667802 std::pair<>::~pair()
[2018-09-05 18:22:37.616][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #46 0x6677d8 __gnu_cxx::new_allocator<>::destroy<>()
[2018-09-05 18:22:37.623][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #47 0x6676c7 std::allocator_traits<>::destroy<>()
[2018-09-05 18:22:37.630][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #48 0x6675d5 std::__detail::_Hashtable_alloc<>::_M_deallocate_node()
[2018-09-05 18:22:37.637][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #49 0x667534 std::__detail::_Hashtable_alloc<>::_M_deallocate_nodes()
[2018-09-05 18:22:37.644][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #50 0x667455 std::_Hashtable<>::clear()
[2018-09-05 18:22:37.652][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #51 0x6673d8 std::_Hashtable<>::~_Hashtable()
[2018-09-05 18:22:37.659][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #52 0x6673b4 std::unordered_map<>::~unordered_map()
[2018-09-05 18:22:37.666][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #53 0x667382 std::pair<>::~pair()
[2018-09-05 18:22:37.674][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #54 0x667358 __gnu_cxx::new_allocator<>::destroy<>()
[2018-09-05 18:22:37.681][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #55 0x667247 std::allocator_traits<>::destroy<>()
[2018-09-05 18:22:37.688][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #56 0x667155 std::__detail::_Hashtable_alloc<>::_M_deallocate_node()
[2018-09-05 18:22:37.695][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #57 0x6670b4 std::__detail::_Hashtable_alloc<>::_M_deallocate_nodes()
[2018-09-05 18:22:37.703][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #58 0x666fd5 std::_Hashtable<>::clear()
[2018-09-05 18:22:37.710][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #59 0x666f58 std::_Hashtable<>::~_Hashtable()
[2018-09-05 18:22:37.717][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #60 0x666f34 std::unordered_map<>::~unordered_map()
[2018-09-05 18:22:37.724][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #61 0x666f02 std::pair<>::~pair()
[2018-09-05 18:22:37.732][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<6216> #62 0x666ed8 __gnu_cxx::new_allocator<>::destroy<>()
[2018-09-05 18:22:37.732][6216][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 6216
Reactions are currently unavailable