Skip to content

kv/kvserver: TestReplicateQueueRebalance failed #73482

@cockroach-teamcity

Description

@cockroach-teamcity

kv/kvserver.TestReplicateQueueRebalance failed with artifacts on master @ 8b696d17241b2eb2d333e23750cf6fdc433897d8:

F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0212cf900, {0x3bb4940, 0x7d8eac0})
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x479
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !runtime/pprof.Do({0x561f368, 0xc000074108}, {{0xc0005d0660, 0xc021bf2090, 0xc007b51f80}}, 0xc007b51fc0)
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/pebble/record.NewLogWriter.func2()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x5c
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/cockroachdb/pebble/record.NewLogWriter
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x329
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !goroutine 59337086 [select]:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func1()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:217 +0x8e
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:213 +0x285
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !goroutine 76054908 [runnable]:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/sasha-s/go-deadlock.lock.func1()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:182
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/sasha-s/go-deadlock.lock
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:182 +0x16f
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !For more context, check log files in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueRebalance2770073773
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !****************************************************************************
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !This node experienced a fatal error (printed above), and as a result the
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !process is terminating.
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !will try to determine the root cause, recommend next steps, and we can
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !improve CockroachDB based on your report.
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !Please submit a crash report by following the instructions here:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !If you would rather not post publicly, please contact us directly at:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !    support@cockroachlabs.com
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !The Cockroach Labs team appreciates your feedback.


ERROR: exit status 7

9 runs completed, 1 failures, over 59m16s
context canceled
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions