-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvserver: raft receive queue may OOM under overload #71805
Copy link
Copy link
Open
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV TeamKV Team
Description
roachtest.restore2TB/nodes=6/cpus=8/pd-volume=2500GB failed with artifacts on release-21.2 @ c5a6b266917ee3846dbd7ae1126c6a5d55cf439b:
| (1) signal: killed
| Error types: (1) *exec.ExitError
Wraps: (5) context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
monitor.go:128,restore.go:449,test_runner.go:777: monitor failure: unexpected node event: 5: dead (exit status 137)
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestore.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:449
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) monitor failure
Wraps: (3) unexpected node event: 5: dead (exit status 137)
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString
cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3613572-1634797350-67-n6cpu8 --oneshot --ignore-empty-nodes: exit status 1 3: 10803
4: 11356
6: 11156
2: 13324
1: 12103
5: dead (exit status 137)
Error: UNCLASSIFIED_PROBLEM: 5: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 5: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
This test on roachdash | Improve this report!
Jira issue: CRDB-10779
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV TeamKV Team