-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: panic due to tracing use after finish #40478
Copy link
Copy link
Closed
Description
Describe the problem
On a testing cluster we observed the following panic several times. It's not clear that anything special was going on. I've deployed a binary which enables DebugUseAfterFinish and will crash when it happen (https://godoc.org/golang.org/x/net/trace#pkg-variables). This issue is to track the panic.
panic: runtime error: index out of range [recovered]
panic: runtime error: index out of range
goroutine 136 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0005d2c60, 0x4e5e460, 0xc000220c00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:181 +0x121
panic(0x3fa4480, 0x77a6c70)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*trace).addEvent(0xc000802340, 0x3f69700, 0xc00868c450, 0xc00c0a0000)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace/trace.go:805 +0x704
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*trace).LazyPrintf(0xc000802340, 0x44a8d46, 0x2, 0xc00c0a8230, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace/trace.go:834 +0x9a
github.com/cockroachdb/cockroach/pkg/util/tracing.(*span).LogFields(0xc00d61d7a0, 0xc00b7f4cc0, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_span.go:393 +0x723
github.com/cockroachdb/cockroach/pkg/util/log.eventInternal(0x4e5e460, 0xc00c886e10, 0xc00c0a0100, 0x450b4ae, 0x24, 0xc0035632e0, 0x2, 0x2)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:128 +0x267
github.com/cockroachdb/cockroach/pkg/util/log.Eventf(...)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:161
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).refreshProposalsLocked(0xc0097f7800, 0x0, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:997 +0x913
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc0097f7800, 0x4e5e460, 0xc0061f2540, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:786 +0xd2e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x4e5e460, 0xc0061f2540, 0xc0097f7800, 0x4e5e460)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3599 +0x131
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000c50000, 0x4e5e460, 0xc0061f2540, 0xc009a4f800, 0xc003563e98, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3352 +0x150
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc000c50000, 0x4e5e460, 0xc000220c00, 0x16)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3588 +0x1f4
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00016e500, 0x4e5e460, 0xc000220c00)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:238 +0x214
Environment:
- CockroachDB at 58d0fc3
- Server OS: Linux
- Client app:
workload kv
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels