Skip to content

roachtest: scaledata/job-coordinator/nodes=3 failed #51700

@cockroach-teamcity

Description

@cockroach-teamcity

(roachtest).scaledata/job-coordinator/nodes=3 failed on provisional_202007220233_v20.2.0-alpha.2@d3119926d33d808c6384cf3e99a7f7435f395489:

The test failed on branch=provisional_202007220233_v20.2.0-alpha.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scaledata/job-coordinator/nodes=3/run_1
	cluster.go:2553,scaledata.go:119,scaledata.go:49,test_runner.go:757: monitor failure: monitor task failed: output in run_043844.694_n4_jobcoordinator_: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107811-1595392378-13-n4cpu4:4 -- ./job-coordinator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.105:26257,10.128.0.140:26257,10.128.0.146:26257' --num_jobs_per_worker=8 --job_period_scale_millis=100 returned: exit status 20
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2541
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2549
		  | main.runSqlapp
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:119
		  | main.registerScaleData.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:49
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2597
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2201
		  | main.runSqlapp.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:115
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2531
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_043844.694_n4_jobcoordinator_
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107811-1595392378-13-n4cpu4:4 -- ./job-coordinator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.105:26257,10.128.0.140:26257,10.128.0.146:26257' --num_jobs_per_worker=8 --job_period_scale_millis=100 returned
		  | stderr:
		  | 2 04:48:46 Transition attempted. currNode: node15 GID: 1308 job_id: job_node8_3 instance_id: 536 status: QUEUED nextStatus: RUNNING txResult: 0
		  | 2020/07/22 04:48:46 Transition attempted. currNode: node15 GID: 1329 job_id: job_node15_6 instance_id: 618 status: QUEUED nextStatus: RUNNING txResult: 0
		  | 2020/07/22 04:48:46 Hit timeout while waiting for jobs to succeed. jobSpecs: [{job_node5_0 500 jobConfig0} {job_node5_1 600 jobConfig1} {job_node5_2 700 jobConfig2} {job_node5_3 800 jobConfig3} {job_node5_4 900 jobConfig4} {job_node5_5 500 jobConfig5} {job_node5_6 600 jobConfig6} {job_node5_7 700 jobConfig7}]
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./job-coordinator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.105:26257,10.128.0.140:26257,10.128.0.146:26257' --num_jobs_per_worker=8 --job_period_scale_millis=100
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /scaledata/job-coordinator/nodes=3

See this test on roachdash
powered by pkg/cmd/internal/issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions