There's a lot of detail in this report. For a summary of this problem, the root cause, and a workaround, see this comment below.
Again trying to reproduce #1130, I found a different issue that caused CockroachDB to exit early:
ATTEMPT 4517
running 1 test
test integration_tests::disks::test_disk_create_disk_that_already_exists_fails ... FAILED
failures:
---- integration_tests::disks::test_disk_create_disk_that_already_exists_fails stdout ----
log file: "/dangerzone/omicron_tmp/try_repro.18644/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.4297.0.log"
note: configured to log to "/dangerzone/omicron_tmp/try_repro.18644/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.4297.0.log"
thread 'integration_tests::disks::test_disk_create_disk_that_already_exists_fails' panicked at 'called `Result::unwrap()` on an `Err` value: Exited', /home/dap/omicron/test-utils/src/dev/mod.rs:141:42
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
integration_tests::disks::test_disk_create_disk_that_already_exists_fails
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 74 filtered out; finished in 0.52s
Test log file:
$ bunyan "/dangerzone/omicron_tmp/try_repro.18644/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.4297.0.log"
[2022-06-02T04:28:27.661727495Z] INFO: test_disk_create_disk_that_already_exists_fails/4297 on ivanova: cockroach temporary directory: /dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb
[2022-06-02T04:28:27.662076216Z] INFO: test_disk_create_disk_that_already_exists_fails/4297 on ivanova: cockroach: copying from seed directory (/home/dap/omicron/target/debug/build/nexus-test-utils-308521ed0d0eed98/out/crdb-base) to storage directory (/dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb/data)
[2022-06-02T04:28:27.67286125Z] INFO: test_disk_create_disk_that_already_exists_fails/4297 on ivanova: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb/data,ballast-size=0 --listen-addr 127.0.0.1:0 --listening-url-file /dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb/listen-url
The CockroachDB output:
$ ls /dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb/
total 9
-rw-r--r-- 1 dap staff 10265 Jun 1 21:28 cockroachdb_stderr
-rw-r--r-- 1 dap staff 0 Jun 1 21:28 cockroachdb_stdout
drwxr-xr-x 4 dap staff 22 Jun 1 21:28 data/
$ cat /dangerzone/omicron_tmp/try_repro.18644/.tmpomCYKb/cockroachdb_stderr
runtime: s.allocCount= 30 s.nelems= 56
fatal error: s.allocCount != s.nelems && freeIndex == s.nelems
goroutine 1 [running, locked to thread]:
runtime.throw(0x5aa10c3, 0x31)
/opt/ooce/go-1.16/src/runtime/panic.go:1117 +0x72 fp=0xc000d6d2f8 sp=0xc000d6d2c8 pc=0x10a9512
runtime.(*mcache).nextFree(0xfffffc7fef180108, 0x316, 0x340, 0xfffffc7fef180108, 0x380)
/opt/ooce/go-1.16/src/runtime/malloc.go:880 +0x228 fp=0xc000d6d330 sp=0xc000d6d2f8 pc=0x107dd88
runtime.mallocgc(0x88, 0x57b9880, 0xfda023cce83e2c01, 0x4)
/opt/ooce/go-1.16/src/runtime/malloc.go:1069 +0x850 fp=0xc000d6d3b8 sp=0xc000d6d330 pc=0x107e5f0
runtime.newobject(...)
/opt/ooce/go-1.16/src/runtime/malloc.go:1177
runtime.mapassign(0x5305d00, 0xc000cb8720, 0x712bea0, 0xc000cb8720)
/opt/ooce/go-1.16/src/runtime/map.go:667 +0x286 fp=0xc000d6d438 sp=0xc000d6d3b8 pc=0x1080286
github.com/aws/aws-sdk-go/aws/endpoints.init()
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go:6880 +0x83005 fp=0xc000d7f7a8 sp=0xc000d6d438 pc=0x479d345
runtime.doInit(0x922b7a0)
/opt/ooce/go-1.16/src/runtime/proc.go:6315 +0xec fp=0xc000d7f8f8 sp=0xc000d7f7a8 pc=0x10b99ec
runtime.doInit(0x9233c80)
/opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc000d7fa48 sp=0xc000d7f8f8 pc=0x10b9972
runtime.doInit(0x9242ae0)
/opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc000d7fb98 sp=0xc000d7fa48 pc=0x10b9972
runtime.doInit(0x92299a0)
/opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc000d7fce8 sp=0xc000d7fb98 pc=0x10b9972
runtime.doInit(0x9277c00)
/opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc000d7fe38 sp=0xc000d7fce8 pc=0x10b9972
runtime.doInit(0x92220c0)
/opt/ooce/go-1.16/src/runtime/proc.go:6292 +0x72 fp=0xc000d7ff88 sp=0xc000d7fe38 pc=0x10b9972
runtime.main()
/opt/ooce/go-1.16/src/runtime/proc.go:208 +0x205 fp=0xc000d7ffe0 sp=0xc000d7ff88 pc=0x10abd25
runtime.goexit()
/opt/ooce/go-1.16/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d7ffe8 sp=0xc000d7ffe0 pc=0x10e35a1
goroutine 52 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x74
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35
goroutine 36 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e1a0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 37 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e270)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 38 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e340)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 39 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e410)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 40 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e4e0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 41 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e5b0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 42 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e680)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 43 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e750)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 44 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e820)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 45 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e8f0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 46 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043e9c0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 47 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043ea90)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 48 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043eb60)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 49 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043ec30)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 50 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043ed00)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 51 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00043edd0)
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173
goroutine 53 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d
goroutine 55 [syscall]:
os/signal.signal_recv(0x0)
/opt/ooce/go-1.16/src/runtime/sigqueue.go:168 +0xa5
os/signal.loop()
/opt/ooce/go-1.16/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/opt/ooce/go-1.16/src/os/signal/signal.go:151 +0x45
goroutine 33 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:165 +0x16b
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0
/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:157 +0x35
This appears to be dying inside the Go runtime memory allocator. It looks like golang/go#45775, which unfortunately was closed last year as not-reproducible.
There's a lot of detail in this report. For a summary of this problem, the root cause, and a workaround, see this comment below.
Again trying to reproduce #1130, I found a different issue that caused CockroachDB to exit early:
Test log file:
The CockroachDB output:
This appears to be dying inside the Go runtime memory allocator. It looks like golang/go#45775, which unfortunately was closed last year as not-reproducible.