Skip to content

bazel: weird test timeout combining --test_filter and stress #77351

@rickystewart

Description

@rickystewart

Internal discussion
TeamCity link

I can reproduce this locally (incl. changes from #76995):

cockroach$ ./dev test pkg/ccl/changefeedccl --stress  --timeout 2m -f TestChangefeedBackfillObservability  
$ bazel test --test_sharding_strategy=disabled pkg/ccl/changefeedccl:all --test_env=GOTRACEBACK=all --test_timeout=180 --run_under '@com_github_cockroachdb_stress//:stress -bazel -shardable-artifacts '"'"'XML_OUTPUT_FILE=/Users/ricky/go/src/github.com/cockroachdb/cockroach/bin/dev-versions/dev.19 merge-test-xmls'"'"' -maxtime=2m0s' --test_filter=TestChangefeedBackfillObservability --test_sharding_strategy=disabled --test_output streamed
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 89c6cdc4-5b20-4da0-b1a4-678e0ec7e8bc
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
INFO: Analyzed 2 targets (1459 packages loaded, 22726 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Deleting stale sandbox base /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/sandbox
38 runs so far, 0 failures, over 5s
84 runs so far, 0 failures, over 10s
129 runs so far, 0 failures, over 15s
182 runs so far, 0 failures, over 20s
230 runs so far, 0 failures, over 25s
279 runs so far, 0 failures, over 30s
329 runs so far, 0 failures, over 35s
381 runs so far, 0 failures, over 40s
430 runs so far, 0 failures, over 45s
480 runs so far, 0 failures, over 50s
533 runs so far, 0 failures, over 55s
582 runs so far, 0 failures, over 1m0s
633 runs so far, 0 failures, over 1m5s
684 runs so far, 0 failures, over 1m10s
734 runs so far, 0 failures, over 1m15s
785 runs so far, 0 failures, over 1m20s
834 runs so far, 0 failures, over 1m25s
885 runs so far, 0 failures, over 1m30s
930 runs so far, 0 failures, over 1m35s
976 runs so far, 0 failures, over 1m40s
1022 runs so far, 0 failures, over 1m45s
1069 runs so far, 0 failures, over 1m50s
1114 runs so far, 0 failures, over 1m55s
1163 runs so far, 0 failures, over 2m0s
1163 runs completed, 0 failures, over 2m0s
-- Test timed out at 2022-03-03 19:49:20 UTC --
SUCCESS

TIMEOUT: //pkg/ccl/changefeedccl:changefeedccl_test (Summary)
      /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/execroot/cockroach/bazel-out/darwin_arm64-fastbuild/testlogs/pkg/ccl/changefeedccl/changefeedccl_test/test.log
INFO: Elapsed time: 191.326s, Critical Path: 182.85s
INFO: 2 processes: 1 internal, 1 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//pkg/ccl/changefeedccl:changefeedccl_test                              TIMEOUT in 180.1s
  /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/execroot/cockroach/bazel-out/darwin_arm64-fastbuild/testlogs/pkg/ccl/changefeedccl/changefeedccl_test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions
ERROR: exit status 3

While we tell stress the maxtime and it does seem to conclude in time, something is stalling and the test is timing out after the 1 minute grace period that we give Bazel to wrap up.

Epic CRDB-8306

Jira issue: CRDB-13543

Metadata

Metadata

Assignees

Labels

A-build-systemC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-dev-inf

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions