-
Notifications
You must be signed in to change notification settings - Fork 4.1k
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [overload,closed ts regressing from X to Y] #61981
Copy link
Copy link
Closed
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.
Description
(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on master@bdff5338ca725bf1cfddf7e3f648bbf02ab42999:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
cluster.go:2688,tpcc.go:785,tpcc.go:617,test_runner.go:767: monitor failure: monitor task failed: failed with output "I210314 10:02:42.292720 1 workload/cli/run.go:361 [-] 1 creating load generator...\nInitializing 5000 connections...\nInitializing 5000 workers and preparing statements...\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: preparing \nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tUPDATE district\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tSET d_next_o_id = d_next_o_id + 1\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tWHERE d_w_id = $1 AND d_id = $2\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tRETURNING d_tax, d_next_o_id: context deadline exceeded\nError: failed to initialize the load generator: preparing \n\t\tUPDATE district\n\t\tSET d_next_o_id = d_next_o_id + 1\n\t\tWHERE d_w_id = $1 AND d_id = $2\n\t\tRETURNING d_tax, d_next_o_id: context deadline exceeded\nError: COMMAND_PROBLEM: exit status 1\n(1) COMMAND_PROBLEM\nWraps: (2) Node 4. Command with error:\n | ```\n | ./cockroach workload run tpcc --warehouses=5000 --workers=5000 --max-rate=736 --wait=false --ramp=25m0s --duration=1h15m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11}\n | ```\nWraps: (3) exit status 1\nError types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError\n": /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2775164-1615705178-42-n12cpu4-geo:4 -- ./cockroach workload run tpcc --warehouses=5000 --workers=5000 --max-rate=736 --wait=false --ramp=25m0s --duration=1h15m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11}: exit status 20
(1) attached stack trace
-- stack trace:
| main.(*monitor).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
| main.(*monitor).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
| main.runTPCCBench
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:785
| main.registerTPCCBenchSpec.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:617
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitor).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.loadTPCCBench
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:710
| [...repeated from below...]
Wraps: (6) failed with output "I210314 10:02:42.292720 1 workload/cli/run.go:361 [-] 1 creating load generator...\nInitializing 5000 connections...\nInitializing 5000 workers and preparing statements...\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: preparing \nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tUPDATE district\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tSET d_next_o_id = d_next_o_id + 1\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tWHERE d_w_id = $1 AND d_id = $2\nE210314 11:04:39.400945 1 workload/cli/run.go:384 [-] 2 +\t\tRETURNING d_tax, d_next_o_id: context deadline exceeded\nError: failed to initialize the load generator: preparing \n\t\tUPDATE district\n\t\tSET d_next_o_id = d_next_o_id + 1\n\t\tWHERE d_w_id = $1 AND d_id = $2\n\t\tRETURNING d_tax, d_next_o_id: context deadline exceeded\nError: COMMAND_PROBLEM: exit status 1\n(1) COMMAND_PROBLEM\nWraps: (2) Node 4. Command with error:\n | ```\n | ./cockroach workload run tpcc --warehouses=5000 --workers=5000 --max-rate=736 --wait=false --ramp=25m0s --duration=1h15m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11}\n | ```\nWraps: (3) exit status 1\nError types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError\n"
Wraps: (7) attached stack trace
-- stack trace:
| main.execCmdWithBuffer
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:566
| main.(*cluster).RunWithBuffer
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2353
| main.loadTPCCBench
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:709
| main.runTPCCBench.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:783
| main.(*monitor).Go.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2666
| golang.org/x/sync/errgroup.(*Group).Go.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1374
Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2775164-1615705178-42-n12cpu4-geo:4 -- ./cockroach workload run tpcc --warehouses=5000 --workers=5000 --max-rate=736 --wait=false --ramp=25m0s --duration=1h15m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11}
Wraps: (9) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *exec.ExitError
More
Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:
-
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #61710 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker
-
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [attempt to create load generator failed] #61181 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [attempt to create load generator failed] C-test-failure O-roachtest O-robot branch-release-20.2
See this test on roachdash
powered by pkg/cmd/internal/issues
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.