Skip to content

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #57062

@cockroach-teamcity

Description

@cockroach-teamcity

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-20.1@982ab5afd472f0f1413711aa196fd405c6a34bd7:

		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:834
		  | [...repeated from below...]
		Wraps: (6) 1 safe detail enclosed
		Wraps: (7) error running tpcc load generator
		Wraps: (8) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:831
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (9) 2 safe details enclosed
		Wraps: (10) output in run_134012.725_n10_workload_run_tpcc
		Wraps: (11) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2468838-1606201600-20-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=910 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=910/stats.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | ion
		  | W201124 13:58:18.066680 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:01:19.893288 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:04:21.785959 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201124 14:07:23.627310 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE district SCATTER": EOF
		  | W201124 14:10:25.592410 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE history SCATTER": driver: bad connection
		  | W201124 14:13:27.682858 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:16:29.952284 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:19:31.863857 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201124 14:22:33.876959 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:25:35.860669 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201124 14:28:37.879126 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:31:39.828426 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:34:41.728017 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201124 14:37:43.932788 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | E201124 14:40:44.012400 1 workload/cli/run.go:379  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | Error: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 10. Command with error:
		  |   | ```
		  |   | ./workload run tpcc --warehouses=2000 --active-warehouses=910 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=910/stats.json
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (12) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *withstack.withStack (9) *safedetails.withSafeDetails (10) *errutil.withMessage (11) *main.withCommandDetails (12) *exec.ExitError

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions