Flaky behavior on MacOS #17437
Description
Description of the bug:
We recently switched from Kokoro to github actions in the protobuf repo. This substantially reduced all our flakes, and there's really only one left that's causing issues. We only see this on Mac/Bazel builds, at rough rate somewhere between 1-5%. We use Bazel 5.1.1 in all our testing. One example of it is here: https://github.com/protocolbuffers/protobuf/actions/runs/4117212221/jobs/7108243227
The symptom is always the same, and looks like a toolchain resolution issue (see below). We have 12 macOS builds in our CI, so even a 1-5% failure rate is a pretty big issue for us.
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Run any bazel build on mac ~30 times and you're likely to see this failure at least once
Which operating system are you running Bazel on?
macOS
What is the output of bazel info release?
development 5.1.1
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
We use the bazelisk installed on our github runners and set USE_BAZEL_VERSION=5.1.1
What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
I found #11520, which looks like a similar issue dating back over 2 years.
Any other information, logs, or outputs that you want to share?
INFO: Repository local_config_cc instantiated at:
/DEFAULT.WORKSPACE.SUFFIX:519:13: in <toplevel>
/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/cc_configure.bzl:184:16: in cc_configure
Repository rule cc_autoconf defined at:
/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/cc_configure.bzl:145:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cc':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
File "/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 211, column 25, in configure_osx_toolchain
_compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
File "/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 136, column 37, in _compile_cc_file
_compile_cc_file_single_arch(repository_ctx, src_name, out_name)
File "/private/var/tmp/_bazel_runner/88067fd1245450c0f2a417255e46b72b/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 83, column 13, in _compile_cc_file_single_arch
fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: