Skip to content

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #57864

@cockroach-teamcity

Description

@cockroach-teamcity

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@960b4cfc54c8b78df56e62f07d7a07b986ceacff:

		  -- stack trace:
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:824
		  | [...repeated from below...]
		Wraps: (6) error running tpcc load generator
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:821
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2626
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (8) output in run_151328.514_n10_workload_run_tpcc
		Wraps: (9) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2515122-1607756763-21-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=1210 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=1210/stats.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | R TABLE customer SCATTER": driver: bad connection
		  | W201212 15:34:35.725321 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | W201212 15:37:37.603978 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201212 15:37:42.368579 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not determine if tables are partitioned: EOF
		  | W201212 15:40:44.334699 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201212 15:43:46.424375 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | W201212 15:46:48.695266 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201212 15:49:50.607559 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | W201212 15:52:52.618407 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201212 15:55:54.594818 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201212 15:58:56.655218 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | W201212 16:01:58.607541 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201212 16:05:00.172228 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201212 16:08:02.375768 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE history SCATTER": EOF
		  | W201212 16:11:04.296977 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | E201212 16:14:04.384111 1 workload/cli/run.go:382  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | Error: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 10. Command with error:
		  |   | ```
		  |   | ./workload run tpcc --warehouses=2000 --active-warehouses=1210 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=1210/stats.json
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (10) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *main.withCommandDetails (10) *exec.ExitError

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions