kvserver: outstanding reproposal at a higher max lease index #97102
Description
Describe the problem
Fatal error on n1 (prob. leaseholder) on this line:
F230214 09:41:18.245640 428 kv/kvserver/replica_application_state_machine.go:238 ⋮ [T1,n1,s1,r155/1:‹/Table/106/1/-7{76224…-57960…}›,raft] 13077 finishing proposal with outstanding reproposal at a higher max lease index: ‹Term:9 Index:13210 Type:EntryNormal Data:"\000dX\340\332\006\356\361\3140\007j\nB\000\222\001\005`\212\002h\002r\227\001\n\224\001\000\000\000\000\000\000\000\000\001\000\000\000\001&\001k\022\362\211\200\224\270\274P{\232h\373\210\000\001txn-\236\023\372a\221\014H\210\204}\017h\202*\217\"\000_\022\004\010\000\020\000\030\000 \000(\0002QF\273B\344\003\n>\n\020\236\023\372a\221\014H\210\204}\017h\202*\217\"\032\014\362\211\200\224\270\274P{\232h\373\210*\n\010\375\320\302\316\276\360\351\241\0270\341\300\016J\n\010\375\320\302\316\276\360\351\241\027P\001*\n\010\260\207\324\253\302\360\351\241\027 \277c\212\001\n\010\263\365\340\216\266\360\351\241\027"›
Could be related to recent changes, e.g. #94633.
To Reproduce
I caught this while running the following on a GCE worker (with master 9c41c48 build):
roachprod create -n3 local
roachprod put local ./cockroach
roachprod start --env "COCKROACH_ROCKSDB_CONCURRENCY=5" --env "COCKROACH_LOG_SST_INFO_TICKS_INTERVAL=6" local
workload run kv --init --histograms=perf/stats.json --concurrency=8196 --splits=100 --duration=30m0s --read-percent=0 --min-block-bytes=1024 --max-block-bytes=1024 --tolerate-errors=true 'postgres://root@127.0.0.1:26257?sslmode=disable'
Jira issue: CRDB-24511