(doesn't upgrade the bazel version we use - it's still 1.0.0 - only uses the latest toolchain).

Just curious, what is the difference?

(doesn't upgrade the bazel version we use - it's still 1.0.0 - only uses the latest toolchain).

Just curious, what is the difference?

If we upgraded bazel version, users might need to upgrade the bazel version too to be able to build.
Updating the toolchain is transparent to the users.

The newer toolchain allows us to use some new definitions - e.g. rbe pool labels https://github.com/grpc/grpc/pull/23087/files#diff-773cbb9cf4d193d10a1d82bed23b4d72L40

Also just curious - might it be possible that an older version of the bazel binary (especially an older major version) won't work with the newer toolchain, in theory?

Also just curious - might it be possible that an older version of the bazel binary (especially an older major version) won't work with the newer toolchain, in theory?

Yes, I'm checking that with the RBE team. So far I got bad news and the new toolchain officially only supports bazel 3.x+. Not sure what that means in terms of bazel versions we can support. (curently we support bazel 1.0.0 and newer, even though we don't test for that explicitly).

The asan/msan/ubsan errors
https://source.cloud.google.com/results/invocations/52280610-a81c-48cc-bbe0-0a54db5d5cc3/targets/%2F%2Ftest%2Fcore%2Fbad_connection:close_fd_test/log

ERROR: /tmpfs/tmp/bazel/external/io_opencensus_cpp/opencensus/trace/BUILD:25:1: Couldn't build file external/io_opencensus_cpp/opencensus/trace/_objs/trace/local_span_store.pic.o: undeclared inclusion(s) in rule '@io_opencensus_cpp//opencensus/trace:trace':
this rule is missing dependency declarations for the following files included by 'external/io_opencensus_cpp/opencensus/trace/internal/local_span_store.cc':
  '/usr/local/lib/clang/11.0.0/share/asan_blacklist.txt'

seem to be the same as
google/oss-fuzz#3093

The rootcause seems to be bazelbuild/bazel#10510

CC @gnossen it seems that both bazel toolchain 2.2.0 and 3.4.2 has its issues (see comment thread for the 2.2.0 problem and the latest test results for 3.4.2 issues)
We might need to upgrade everything to the latest bazel and toolchain (after fixing our builds), but we will also need bazel distribtests to ensure (at least some) compatibility with bazel 1.x and 2.x - suddenly the bazel distribtests seem more of a priority.

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 30 days. It will be closed automatically if no further update occurs in 7 day. Thank you for your contributions!

Superseded by #25001