stability: index out of range on (*TxnMeta).MarshalTo #5998
Description
build sha: c382dd7
brand new beta cluster with photos and block-writer (concurrency=5).
Ran for less than an hour before seeing logs of errors along the lines of:
W160412 10:12:55.732779 storage/replica.go:1433 failed to lookup sender replica 2 in group 80: storage/replica.go:658: replica 2 not found in range 80
I stopped block_writers to start poking around, when node 1 (ec2-54-209-69-52.compute-1.amazonaws.com) crashed with:
panic: runtime error: index out of range
1: running [Created by grpc.(*Server).serveStreams.func1 @ server.go:324]
panic.go:464 panic(0x163ce80, 0xc82000c050)
roachpb data.pb.go:934 (*TxnMeta).MarshalTo(#10, 0xc8250144d8, 0x1c, 0x1c, 0x64, 0, 0)
roachpb data.pb.go:975 (*Transaction).MarshalTo(#10, 0xc8250144d6, 0x1e, 0x1e, 0xc0, 0, 0)
roachpb errors.pb.go:1348 (*Error).MarshalTo(#8, 0xc8250144b6, 0x3e, 0x3e, 0xf7, 0, 0)
roachpb api.pb.go:3847 (*BatchResponse_Header).MarshalTo(#11, 0xc8250144b3, 0x41, 0x41, 0x106, 0, 0)
roachpb api.pb.go:3808 (*BatchResponse).MarshalTo(#11, #9, 0x44, 0x44, 0x44, 0, 0)
roachpb api.pb.go:3793 (*BatchResponse).Marshal(#11, #9, 0x44, 0x44, 0, 0)
proto encode.go:225 Marshal(0x7f2d22317380, #11, 0, 0, 0, 0, 0)
grpc rpc_util.go:70 protoCodec.Marshal(#1, #11, 0, 0, 0, 0, 0)
grpc <autogenerated>:18 (*protoCodec).Marshal(#3, #1, #11, 0, 0, 0, 0, 0)
grpc rpc_util.go:248 encode(#14, #3, #1, #11, 0, 0, 0, 0, 0, 0, ...)
grpc server.go:412 (*Server).sendResponse(#6, #13, #4, #12, #1, #11, 0, 0, 0xc833ea7f08, 0, ...)
grpc server.go:526 (*Server).processUnaryRPC(#6, #13, #4, #12, #5, #2, 0, 0, 0)
grpc server.go:646 (*Server).handleStream(#6, #13, #4, #12, 0)
grpc server.go:323 (*Server).serveStreams.func1.1(#7, #6, #13, #4, #12)
Node log:
node1.log.parse.txt