In a recent customer escalation, we saw a newly elected Raft leader spend so long to send 100 log entries to catch up a follower that it missed its heartbeat. The log entries were found to be about 132 kb each, and we limit these messages to be a maximum of 32 kb (but at least 1 entry), so responses were being sent with a single entry each. It seemed to be taking 60 ms to retrieve a single entry, totalling 6s for 100 log entries. The log was also huge, at ~900,000 entries, with the current commit index at 1500.

Digging into the Raft code, we find that raft.maybeSendAppend() calls int raftLog.entries() to fetch entries to send to the follower. Notice how it limits this solely by maxsize (32 kb), otherwise using the entire Raft log (lastIndex()+1) as an upper bound:

https://github.com/cockroachdb/vendored/blob/310ea3cd09b55055b290f0a4cf9991a91725797e/go.etcd.io/etcd/raft/v3/log.go#L287-L292

This in turn gets passed into Storage.Entries():

https://github.com/cockroachdb/vendored/blob/310ea3cd09b55055b290f0a4cf9991a91725797e/go.etcd.io/etcd/raft/v3/log.go#L350

Our implementation of Entries(), however, does not pass the max size bound to the MVCC iterator, only the key bounds, instead relying on the passed scan function to error when the size limit is exceeded:

The problem is that MVCCIterate will actually fetch 1000 entries before calling scanFunc:

Since we're using the entire Raft log as an upper bound, not passing in a MVCCScanOptions.MaxSize for the iterator, and only doing size limiting in the passed scanFunc, this means that every time we fetch a single log entry we're actually fetching and decoding 1000 entries, each 132 kb large.

The simplest fix here is probably to just pass in the max size from Raft in MVCCScanOptions.

/cc @cockroachdb/kv