@philwo I'm back... I tried out a recent build of bazel on our code base, and, after hacking around some changes to how the toolchains have changed, ran into issues. I saw 100% CPU usage (1 core) for Java, and, after some progress, no build progress. I waited 40 minutes with no progress (Bazel wasn't reporting any actions were being executed). atop said that bazel was writing 4 MB/s to disk at the time.

Attaching with strace yielded the following info:

[pid 58046] sched_yield() = 0
[pid 59244] lstat("/home/austin/.cache/bazel/_bazel_austin/dd758a6cc6943296d5c6a65e2c252c63/bazel-sandbox/202e6024-7d68-4028-8ba9-65ade44518e7-240/peloton-tech/execroot/peloton-tech/external/androidndk/ndk/platforms/android-21/arch-arm/usr/include/linux/netfilter/ipset/ip_set_bitmap.h", <unfinished ...>
[pid 58046] write(129, "l", 1 <unfinished ...>
[pid 59244] <... lstat resumed> {st_mode=S_IFLNK|0777, st_size=198, ...}) = 0
[pid 58046] <... write resumed> ) = 1
[pid 59244] unlink("/home/austin/.cache/bazel/_bazel_austin/dd758a6cc6943296d5c6a65e2c252c63/bazel-sandbox/202e6024-7d68-4028-8ba9-65ade44518e7-240/peloton-tech/execroot/peloton-tech/external/androidndk/ndk/platforms/android-21/arch-arm/usr/include/linux/netfilter/ipset/ip_set_bitmap.h" <unfinished ...>
[pid 58046] futex(0x7fd544073094, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd544073090, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 0
[pid 56541] futex(0x7fd544073094, FUTEX_WAIT_PRIVATE, 1631649, NULL <unfinished ...>
[pid 58046] sched_yield() = 0
[pid 56541] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 58046] sched_yield() = 0
[pid 58046] futex(0x7fd47c004e04, FUTEX_WAIT_PRIVATE, 8519, NULL <unfinished ...>
[pid 56541] futex(0x7fd544073068, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 56541] futex(0x7fd47c004e04, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd47c004e00, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid 58046] <... futex resumed> ) = 0
[pid 58046] futex(0x7fd47c004dd8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 56541] <... futex resumed> ) = 1
[pid 56541] futex(0x7fd47c004dd8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 58046] <... futex resumed> ) = 0
[pid 58046] futex(0x7fd47c004dd8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 56541] <... futex resumed> ) = 1
[pid 58046] write(129, "i", 1) = 1
[pid 58046] futex(0x7fd544073094, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd544073090, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 0
[pid 56541] futex(0x7fd544073094, FUTEX_WAIT_PRIVATE, 1631651, NULL <unfinished ...>
[pid 58046] sched_yield() = 0
[pid 56541] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 58046] sched_yield() = 0
[pid 58046] futex(0x7fd47c004e04, FUTEX_WAIT_PRIVATE, 8521, NULL <unfinished ...>
[pid 56541] futex(0x7fd544073068, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 56541] futex(0x7fd47c004e04, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd47c004e00, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid 58046] <... futex resumed> ) = 0
[pid 58046] futex(0x7fd47c004dd8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 56541] <... futex resumed> ) = 1
[pid 56541] futex(0x7fd47c004dd8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 58046] <... futex resumed> ) = 0

I unfortunately don't have a repo I can give you to test with. I'm trying to get the robotics code building with the latest bazel since that is open source to see if the issue persists, but I'm running into the same toolchain issues, and I've been dragged away from my desk enough that I'm not making fast progress.