Something is wrong with "Basic Tests C/C++ MacOS" test since Jul 28 #23661
Description
I am not sure what exactly is wrong, but vast majority of the "Basic Tests C/C++ MacOS" PR tests failed today.
Kokoro job: grpc/core/pull_request/macos/grpc_basictests_c_cpp
Vast majority of builds start failing since about 10am PDT Jul 28, 2020.
A few runs failed with the following error:
2020-07-28 13:41:13,502 FAILED: bins/opt/client_callback_end2end_test --gtest_filter=ClientCallbackEnd2endTest/ClientCallbackEnd2endTest.SequentialGenericRpcsAsBidiWithReactorReuse/6 GRPC_POLL_STRATEGY=poll
D0728 13:38:55.396281000 140735574311808 ev_posix.cc:173] Using polling engine: poll
D0728 13:38:55.397595000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "grpclb"
D0728 13:38:55.397632000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "cds_experimental"
D0728 13:38:55.397658000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "eds_experimental"
D0728 13:38:55.397677000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "lrs_experimental"
D0728 13:38:55.397700000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "priority_experimental"
D0728 13:38:55.397718000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "weighted_target_experimental"
D0728 13:38:55.397744000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "xds_routing_experimental"
D0728 13:38:55.397769000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "pick_first"
D0728 13:38:55.397788000 140735574311808 lb_policy_registry.cc:40] registering LB policy factory for "round_robin"
D0728 13:38:55.397811000 140735574311808 dns_resolver_ares.cc:507] Using ares dns resolver
[waiter.cc : 223] RAW: pthread_mutex_destroy failed: 16
Another common failure seems to be TIMEOUT: bins/opt/async_end2end_test (this error seems to be most common):
2020-07-28 19:09:58,743 TIMEOUT: bins/opt/async_end2end_test --gtest_filter=AsyncEnd2end/AsyncEnd2endTest.SimpleClientStreamingWithCoalescingApi/8 GRPC_POLL_STRATEGY=poll [pid=28339, time=302.2sec]
2020-07-28 19:09:58,777 FAILED: bins/opt/async_end2end_test --gtest_filter=AsyncEnd2end/AsyncEnd2endTest.SimpleClientStreamingWithCoalescingApi/8 GRPC_POLL_STRATEGY=poll
A few runs failed super early with:
+ echo 'Running run_tests.py in workspace workspace_c_macos_dbg_native'
Running run_tests.py in workspace workspace_c_macos_dbg_native
+ python workspace_c_macos_dbg_native/tools/run_tests/run_tests.py -t -j 4 -x ../run_tests/c_macos_dbg_native/sponge_log.xml --report_suite_name c_macos_dbg_native -l c -c dbg --iomgr_platform native --max_time 3600
2020-07-28 16:22:03,420 START: make
2020-07-28 16:22:05,058 Makefile:13686: *** multiple target patterns. Stop.
2020-07-28 16:22:05,058 FAILED: make [ret=2, pid=4652, time=1.6sec]
2020-07-28 16:22:05,058 FAILED: Some tests failed
Another common error seems to be that it just said "Tool failed":
https://source.cloud.google.com/results/invocations/a923e956-6914-4ca5-aea5-cfa4e797b42a/targets