Looks like it's still not fixed yet: https://buildkite.com/bazel/bazel-at-head-plus-disabled/builds/1550#0185dd3c-a7a5-48cc-b53d-e479e7b0ec18

Ah hm. Do you have any tips to repro this? Running locally, things are fine. Our CI is also OK afaik.

Aha, ok, I think I was able to repro by manually deleting the bazel binary after bazel starts the test, but before the test starts bazel as a subprocess.

The CI sets USE_BAZEL_VERSION=/some/tmp/build/bazel in the environment. This gets passed along via the --test_env=USE_BAZEL_VERSION flag to the CI's (outer) bazel invocation. Later, when run_acceptance_test.py in rules_python tries to run bazel as a subprocess, bazelisk tries to copy /some/tmp/build/bazel to its cache directory. But, for some reason, that file is gone. Or maybe it isn't gone, but not readable by whatever context the tests are running in?

That the path USE_BAZEL_VERSION is pointing to doesn't exist anymore is a bit weird, especially because the outer bazel invocation passes --test_env=USE_BAZEL_VERSION, which would imply it's expected that tests need to propagate the value to invoke bazel themselves.

So, why would /tmp/whatever/bazel path disappear? Is this something sandboxes would cause?

Aha! This is because of --sandbox_tmpfs_path=/tmp hiding USE_BAZEL_VERSION=/tmp/somepath

Would it be possible to adjust either of those?

do you know where --sandbox_tmpfs_path=/tmp is being specified?

Github search showed this commit:
dc6796a

But that's all I can find. There's a couple issues mentioning it, but they none of them seemed relevant.

Aha, maybe this is it: https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/bazelci.py#L1877

Ok, I figured out a workaround in the tests. PTAL