stability: refcount replicas to avoid races/use-after-destroy #8630
Description
Running block_writer against a local 4 node cluster I found this gem in the logs:
W160817 19:56:21.031906 storage/store.go:2442 store=4:4: raft ready processing: 19.8s
A few lines earlier we have:
I160817 19:56:12.520081 storage/store.go:2743 store 4 sstables (read amplification = 1):
6 [ 80M 1 ]: 80M
I160817 19:56:21.012972 storage/replica_trigger.go:308 store=4:4 range=1 [/Min-/Table/11): new range lease replica {4 4 3} 2016-08-17 23:55:57.278845676 +0000 UTC 5.300133463s following replica {1 1 1} 2016-08-17 23:55:44.665433835 +0000 UTC 12.613411841s [physicalTime=2016-08-17 23:56:21.012929761 +0000 UTC]
That's a 9s jump between 2 log lines. The last raft ready message was:
I160817 19:56:01.270414 storage/raft.go:150 store=4:4 range=5 [/Table/14-/Table/50) raft ready
Outgoing Message[0]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[1]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[2]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[3]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[4]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[5]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[6]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[7]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[8]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[9]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
Outgoing Message[10]: 4->2 MsgHeartbeatResp Term:7 Log:0/0
That looks innocuous enough. Where did the time go?