stability: unexpected panic: llrb: inverted range #6027
Description
Build sha: 08d6640
Beta cluster has been having issues, so some of them may be the direct cause of this (eg: #5998, #6000, #6020)
I just restarted node 4 (ec2-52-91-234-29.compute-1.amazonaws.com) with a recent sha (08d6640). About 15s later, node 3 (ec2-54-209-150-36.compute-1.amazonaws.com) crashed.
Node 4 shows the following:
I160413 10:46:45.889824 http2_client.go:881 transport: http2Client.notifyError got notified that the client transport was broken EOF.
Node 3 died with:
F160413 10:46:45.783176 util/log/log.go:42 unexpected panic: llrb: inverted range
1: running [Created by http2.(*serverConn).processHeaders @ server.go:1444]
log clog.go:858 getStacks(0xecea01a01, 0, 0, 0)
log clog.go:724 (*loggingT).outputLogEntry(#4, 0x3, 0x1f705bf, 0xf, 0x2a, 0xc8230fa630, 0x26)
log structured.go:57 addStructured(0, 0, 0x3, 0x2, #3, 0x14, #24, 0x1, 0x1)
log log.go:73 logDepth(0, 0, 0x1, 0x3, #3, 0x14, #24, 0x1, 0x1)
log log.go:179 Fatalf(#3, 0x14, #24, 0x1, 0x1)
log log.go:42 FatalOnPanic()
panic.go:426 panic(0x140ef80, 0xc82bc25900)
llrb llrb.go:505 (*Tree).DoRange(0xc820405478, 0xc826b5d220, #33, 0xc8230fa570, #33, 0xc8230fa5a0, 0x18)
cache cache.go:370 (*OrderedCache).DoRange(0xc820405420, 0xc826b5d2d0, #1, 0xc822113080, #1, 0xc8221130a0)
kv range_cache.go:617 (*rangeDescriptorCache).clearOverlappingCachedRangeDescriptors(#15, #20, 0, 0)
kv range_cache.go:561 (*rangeDescriptorCache).insertRangeDescriptorsLocked(#15, #20, 0x1, 0x1, 0, 0)
kv range_cache.go:217 insertRangeDescriptorsLocked)-fm(#20, 0x1, 0x1, 0, 0)
kv range_cache.go:244 (*evictionToken).EvictAndReplace(0xc826b5dcd8, #20, 0x1, 0x1, 0, 0)
kv dist_sender.go:761 (*DistSender).sendChunk(#12, #32, #23, #35, 0, 0, 0, 0, 0, 0, ...)
kv dist_sender.go:546 (*DistSender).Send(#12, #32, #23, #35, 0, 0, 0, 0, 0, 0, ...)
kv txn_coord_sender.go:409 (*TxnCoordSender).Send(#13, #32, #23, 0, 0, 0, 0, 0, 0, 0, ...)
client db.go:474 (*DB).send(#16, 0, 0x2, #25, 0x1, 0x1, 0x25, 0x40)
client db.go:421 send)-fm(0, 0x2, #25, 0x1, 0x1, 0xc82ca11fb0, 0x25)
client db.go:386 sendAndFill(0xc826b5efc8, #19, 0x25, 0x25)
client db.go:421 (*DB).RunWithResponse(#16, #19, #2, #26)
client db.go:412 (*DB).Run(#16, #19, #19)
client db.go:490 runOneResult(#34, #16, #19, 0, 0, 0, 0, 0, 0, 0, ...)
client db.go:293 (*DB).scan(#16, #2, #26, #2, #27, 0, #6, 0, 0, 0, ...)
client db.go:309 (*DB).ScanInconsistent(#16, #2, #26, #2, #27, 0, 0, 0, 0, 0)
ts query.go:435 (*DB).Query(#7, 0xc826235920, 0x19, 0xc82a826a68, 0xc82a826a74, 0xc82a826a80, #5, 0, 0, 0x1, ...)
ts server.go:84 (*Server).handleQuery(#11, #30, #29, #28, 0, 0, 0)
ts server.go:49 handleQuery)-fm(#30, #29, #28, 0, 0, 0)
httprouter router.go:344 (*Router).ServeHTTP(#8, #30, #29, #28)
ts server.go:55 (*Server).ServeHTTP(#11, #30, #29, #28)
http server.go:1910 (*ServeMux).ServeHTTP(#14, #30, #29, #28)
server server.go:440 (*Server).ServeHTTP(#9, #30, #29, #28)
http server.go:2081 serverHandler.ServeHTTP(#10, #31, #22, #28)
http server.go:2489 initNPNRequest.ServeHTTP(#17, #10, #31, #22, #28)
http <autogenerated>:253 (*initNPNRequest).ServeHTTP(#21, #31, #22, #28)
http h2_bundle.go:3847 (Handler).ServeHTTP-fm(#31, #22, #28)
http2 server.go:1679 (*serverConn).runHandler(#18, #22, #28, 0xc8262352e0)
Node 3 log:
node3.log.parse.txt.gz
Node 4 log:
node4.log.parse.txt
I'll see if I can add debugging here and try again.