Skip to content

[Bazel CI] Tests are slow and flaky on macOS arm64  #23726

@meteorcloudy

Description

@meteorcloudy

//src/test/shell/bazel:bazel_bootstrap_distfile_tar_test and //src/test/shell/bazel:bazel_determinism_test are extremely slow:
https://buildkite.com/bazel/google-bazel-presubmit/builds/84154#01922292-cca6-4e56-939b-05a5c7b59da1

Flaky tests are very frequent:
https://buildkite.com/bazel/google-bazel-presubmit/builds/84157#019222c1-653c-4a75-bdf7-7f76d5937946

//src/test/py/bazel:cc_import_test                                        FLAKY, failed in 1 out of 2 in 69.9s
  Stats over 2 runs: max = 69.9s, min = 55.5s, avg = 62.7s, dev = 7.2s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/py/bazel/cc_import_test/test_attempts/attempt_1.log
//src/test/shell/integration:config_stripped_outputs_test                 FLAKY, failed in 1 out of 2 in 126.8s
  Stats over 2 runs: max = 126.8s, min = 78.5s, avg = 102.6s, dev = 24.2s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/config_stripped_outputs_test/test_attempts/attempt_1.log
//src/test/shell/bazel:bazel_rules_java_override_test                     FLAKY, failed in 2 out of 3 in 52.4s
  Stats over 3 runs: max = 52.4s, min = 8.7s, avg = 31.2s, dev = 17.9s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/bazel_rules_java_override_test/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/bazel_rules_java_override_test/test_attempts/attempt_2.log
//src/test/shell/bazel:build_files_test                                   FLAKY, failed in 2 out of 3 in 49.8s
  Stats over 3 runs: max = 49.8s, min = 25.1s, avg = 41.6s, dev = 11.6s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/build_files_test/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/build_files_test/test_attempts/attempt_2.log
//src/test/shell/bazel:bazel_coverage_java_jdk21_toolchain_released_test FAILED in 3 out of 3 in 248.6s
  Stats over 3 runs: max = 248.6s, min = 199.0s, avg = 221.1s, dev = 20.6s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/bazel_coverage_java_jdk21_toolchain_released_test/test.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/bazel_coverage_java_jdk21_toolchain_released_test/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/bazel/bazel_coverage_java_jdk21_toolchain_released_test/test_attempts/attempt_2.log
//src/test/shell/integration:test_test                                   FAILED in 3 out of 3 in 216.0s
  Stats over 3 runs: max = 216.0s, min = 158.5s, avg = 191.3s, dev = 24.2s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/test_test/test.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/test_test/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/test_test/test_attempts/attempt_2.log
//src/test/java/com/google/devtools/build/lib/rules/config:ConfigRulesTests PASSED in 34.3s
  Stats over 5 runs: max = 34.3s, min = 17.8s, avg = 25.3s, dev = 7.0s
//src/test/shell/integration:bazel_sandboxed_worker_test                 FAILED in 6 out of 7 in 146.6s
  Stats over 7 runs: max = 146.6s, min = 98.7s, avg = 125.7s, dev = 15.4s
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_1_of_3/test.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_1_of_3/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_1_of_3/test_attempts/attempt_2.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_2_of_3/test.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_2_of_3/test_attempts/attempt_1.log
  /private/var/tmp/_bazel_buildkite/00e02099ed8d75d374b9c12be02eaf4c/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/src/test/shell/integration/bazel_sandboxed_worker_test/shard_2_of_3/test_attempts/attempt_2.log
//src/test/java/com/google/devtools/build/lib/query2/engine:AllTests     PASSED in 16.9s
  Stats over 10 runs: max = 16.9s, min = 6.2s, avg = 11.1s, dev = 3.1s
//src/test/shell/bazel/remote:remote_execution_test                      FAILED in 9 out of 12 in 256.6s

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1I'll work on this now. (Assignee required)breakageteam-OSSIssues for the Bazel OSS team: installation, release processBazel packaging, websitetype: bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions