make roachprod-stress PKG=./pkg/kv/kvnemesis TESTS=TestKVNemesisMultiNode TESTTIMEOUT=2m CLUSTER=nathan-stress
panic: resolved timestamp 1613982715.406343041,0 equal to or above timestamp of operation {<nil> txn_id:2a60432a-fa66-408c-9160-7ff86c54f2b9 txn_key:"\272\0220b2e8742\000\001" txn_min_timestamp:<wall_time:1613982715404791290 > timestamp:<wall_time:1613982715404791290 > <nil> <nil> <nil> <nil>}
goroutine 2815 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000914380, 0x5118be0, 0xc004067e30)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x3bca140, 0xc0045ff4c0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*resolvedTimestamp).assertOpAboveRTS(0xc004dfcfd8, 0x0, 0xc003334780, 0x0, 0x0, 0x0, 0x0, 0x166604ba7952adfa, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/resolved_timestamp.go:251 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*resolvedTimestamp).consumeLogicalOp(0xc004dfcfd8, 0x0, 0xc003334780, 0x0, 0x0, 0x0, 0x0, 0xc0032b6060)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/resolved_timestamp.go:146 +0x296
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*resolvedTimestamp).ConsumeLogicalOp(0xc004dfcfd8, 0x0, 0xc003334780, 0x0, 0x0, 0x0, 0x0, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/resolved_timestamp.go:132 +0x4d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).consumeLogicalOps(0xc004dfcf20, 0x5118b20, 0xc003ef8b00, 0xc0042d2990, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:591 +0x227
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).consumeEvent(0xc004dfcf20, 0x5118b20, 0xc003ef8b00, 0xc003518230)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:540 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc004dfcf20, 0x5118be0, 0xc004067e30, 0xc0042b3820, 0xc000914380)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:286 +0xb7a
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1(0x5118be0, 0xc004067e30)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:196 +0x50
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000914380, 0x5118be0, 0xc004067e30, 0x0, 0xc0042b3840)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
After about 45 minutes of running kvnemesis, I often see an error like:
At first, I figured that this was fallout from #59566, but I've managed to reproduce with that change disabled. This is an
MVCCWriteIntentOpbelow a range's resolved timestamp, which should never be allowed.