Skip to content

storage: panic due to tracing use after finish #40478

@ajwerner

Description

@ajwerner

Describe the problem

On a testing cluster we observed the following panic several times. It's not clear that anything special was going on. I've deployed a binary which enables DebugUseAfterFinish and will crash when it happen (https://godoc.org/golang.org/x/net/trace#pkg-variables). This issue is to track the panic.

panic: runtime error: index out of range [recovered]
       panic: runtime error: index out of range
goroutine 136 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0005d2c60, 0x4e5e460, 0xc000220c00)
       /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:181 +0x121
panic(0x3fa4480, 0x77a6c70)
       /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*trace).addEvent(0xc000802340, 0x3f69700, 0xc00868c450, 0xc00c0a0000)
       /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace/trace.go:805 +0x704
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*trace).LazyPrintf(0xc000802340, 0x44a8d46, 0x2, 0xc00c0a8230, 0x1, 0x1)
       /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace/trace.go:834 +0x9a
github.com/cockroachdb/cockroach/pkg/util/tracing.(*span).LogFields(0xc00d61d7a0, 0xc00b7f4cc0, 0x1, 0x1)
       /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_span.go:393 +0x723
github.com/cockroachdb/cockroach/pkg/util/log.eventInternal(0x4e5e460, 0xc00c886e10, 0xc00c0a0100, 0x450b4ae, 0x24, 0xc0035632e0, 0x2, 0x2)
       /go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:128 +0x267
github.com/cockroachdb/cockroach/pkg/util/log.Eventf(...)
       /go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:161
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).refreshProposalsLocked(0xc0097f7800, 0x0, 0x1)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:997 +0x913
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc0097f7800, 0x4e5e460, 0xc0061f2540, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:786 +0xd2e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x4e5e460, 0xc0061f2540, 0xc0097f7800, 0x4e5e460)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3599 +0x131
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000c50000, 0x4e5e460, 0xc0061f2540, 0xc009a4f800, 0xc003563e98, 0x0)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3352 +0x150
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc000c50000, 0x4e5e460, 0xc000220c00, 0x16)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3588 +0x1f4
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00016e500, 0x4e5e460, 0xc000220c00)
       /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:238 +0x214

Environment:

  • CockroachDB at 58d0fc3
  • Server OS: Linux
  • Client app: workload kv

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions