-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Closed
Labels
Description
bazel test --runs_per_test=100 //test/integration:hds_integration_test fails 7 out of 100.
A sample stack trace:
[==========] Running 20 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 20 tests from IpVersions/HdsIntegrationTest
[ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointHealthyHttp/IPv4
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointHealthyHttp/IPv4 (1098 ms)
[ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointHealthyHttp/IPv6
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointHealthyHttp/IPv6 (1069 ms)
[ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutHttp/IPv4
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutHttp/IPv4 (1085 ms)
[ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutHttp/IPv6
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutHttp/IPv6 (1068 ms) [ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointUnhealthyHttp/IPv4
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointUnhealthyHttp/IPv4 (1072 ms) [ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointUnhealthyHttp/IPv6
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointUnhealthyHttp/IPv6 (1073 ms) [ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutTcp/IPv4
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutTcp/IPv4 (1063 ms) [ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutTcp/IPv6
[ OK ] IpVersions/HdsIntegrationTest.SingleEndpointTimeoutTcp/IPv6 (1063 ms) [ RUN ] IpVersions/HdsIntegrationTest.SingleEndpointHealthyTcp/IPv4
[2018-08-13 22:15:51.540][18244][critical][assert] test/integration/fake_upstream.h:302] assert failure: parented_ || allow_unexpected_disconnects_. Details: An queued upstream connection was torn down without being associated with a fake connection. Either manage the connection via waitForRawConnection() or waitForHttpConnection(), or set_allow_unexpected_disconnects(true).
See https://github.com/envoyproxy/envoy/blob/master/test/integration/README.md#unparented-upstream-connections [2018-08-13 22:15:51.540][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0x3e9000042b3
[2018-08-13 22:15:51.540][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<18244> obj</lib/x86_64-linux-gnu/libc.so.6> (If unsymbolized, use tools/stack_decode.py):
[2018-08-13 22:15:51.541][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<18244> #0 0x7fbbb441ffff (unknown) [2018-08-13 22:15:51.541][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<18244> #1 0x7fbbb4421429 (unknown) [2018-08-13 22:15:51.541][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<18244> obj</home/zlizan/.cache/bazel/_bazel_zlizan/48603501217cff46cc0f43e298b8b8e9/sandbox/processwrapper-sandbox/1523/execroot/envoy/bazel-out/k8-fastbuild/bin/test/integration/hds_integration_test.runfiles/envoy/test/integration/hds_integration_test> [2018-08-13 22:15:51.554][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #2 0x644731 Envoy::QueuedConnectionWrapper::QueuedConnectionWrapper()::{lambda()#1}::operator()()
[2018-08-13 22:15:51.568][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #3 0x64448c std::_Function_handler<>::_M_invoke() [2018-08-13 22:15:51.581][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #4 0x62d30d std::function<>::operator()()
[2018-08-13 22:15:51.595][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #5 0x62d1b7 Envoy::Common::CallbackManager<>::runCallbacks() [2018-08-13 22:15:51.608][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #6 0x62cc34 Envoy::SharedConnectionWrapper::onEvent() [2018-08-13 22:15:51.621][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #7 0x758845 Envoy::Network::ConnectionImpl::raiseEvent()
[2018-08-13 22:15:51.635][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #8 0x757681 Envoy::Network::ConnectionImpl::closeSocket()
[2018-08-13 22:15:51.648][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #9 0x759fc5 Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-13 22:15:51.662][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #10 0x75c13d Envoy::Network::ConnectionImpl::ConnectionImpl()::$_2::operator()()
[2018-08-13 22:15:51.675][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #11 0x75bff0 std::_Function_handler<>::_M_invoke() [2018-08-13 22:15:51.688][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #12 0x74f1c9 std::function<>::operator()() [2018-08-13 22:15:51.702][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #13 0x74ed88 Envoy::Event::FileEventImpl::assignEvents()::$_0::operator()() [2018-08-13 22:15:51.716][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #14 0x74ebb7 Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke()
[2018-08-13 22:15:51.729][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #15 0x1848198 event_process_active_single_queue [2018-08-13 22:15:51.742][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #16 0x1844c2f event_base_loop
[2018-08-13 22:15:51.755][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #17 0x749afc Envoy::Event::DispatcherImpl::run() [2018-08-13 22:15:51.769][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #18 0x6387e5 Envoy::FakeUpstream::threadRoutine()
[2018-08-13 22:15:51.782][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #19 0x63c877 Envoy::FakeUpstream::FakeUpstream()::$_9::operator()() [2018-08-13 22:15:51.795][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #20 0x63c72c std::_Function_handler<>::_M_invoke()
[2018-08-13 22:15:51.795][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #21 0x62d30d std::function<>::operator()() [2018-08-13 22:15:51.808][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #22 0x153d2eb Envoy::Thread::Thread::Thread()::$_0::operator()()
[2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #23 0x153d2c7 Envoy::Thread::Thread::Thread()::$_0::__invoke() [2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<18244> obj</lib/x86_64-linux-gnu/libpthread.so.0>
[2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<18244> #24 0x7fbbb5030493 start_thread [2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<18244> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<18244> #25 0x7fbbb44d5ace (unknown)
[2018-08-13 22:15:51.821][18244][critical][backtrace] bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 18244
Reactions are currently unavailable