Skip to content

roachtest: tpce/c=5000/nodes=3 failed #66719

@cockroach-teamcity

Description

@cockroach-teamcity

roachtest.tpce/c=5000/nodes=3 failed with artifacts on master @ 1b686aef9949c1c7ef930b55bd1fbc0ed2e8268a:

		
		thread '<unnamed>' panicked at 'called ``Result::unwrap()`` on an ``Err`` value: "SendError(..)"', driver/src/customer_emulator.rs:34:58
		thread '<unnamed>' panicked at 'called ``Result::unwrap()`` on an ``Err`` value: "SendError(..)"', driver/src/market_exchange_emulator.rs:48:53
		thread '<unnamed>' panicked at 'called ``Result::unwrap()`` on an ``Err`` value: "SendError(..)"', driver/src/data_maintenance_emulator.rs:54:30

	cluster.go:2484,tpce.go:100,tpce.go:117,test_runner.go:757: monitor failure: unexpected node event: 1: dead (exit status 134)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2472
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2480
		  | main.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpce.go:100
		  | main.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpce.go:117
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 1: dead (exit status 134)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1328,context.go:89,cluster.go:1317,test_runner.go:838: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3103795-1624342309-74-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead (exit status 134)
		3: 10484
		2: 10280
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 134)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1154
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:276
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2051
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 1: dead (exit status 134)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh tpce/c=5000/nodes=3

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Metadata

Metadata

Assignees

Labels

C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-test-failureBroken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions