In a support case, we saw that an assertion like the following could be hit during inline (timestamp == 0) DeleteRange evaluation:

F230412 18:36:44.436420 52096636 kv/kvserver/pkg/kv/kvserver/replica_batch_updates.go:227 ⋮ [n1,s1,r774/1:‹/System/tsd/cr.store.{qu…-re…}›] 7351 trying to bump to 1681309128.304176583,1 <= ba.Timestamp: 1681324603.939164946,0

We suspect that this is due to corruption that allowed the DeleteRange to see an MVCC value (timestamp != 0).

Inline DeleteRange requests are incompatible with MVCC values, but the presence of these values should not lead to a crash. Instead, they should return an error:

Our leading theory is that the corruption led to an unexpected WriteTooOld error because the DeleteRange's search phase sets the MVCCScanOptions.FailOnMoreRecent=true scan option:

We can demonstrate this with the following test:

func TestMVCCDeleteRangeInlineEncountersMVCCVersion(t *testing.T) {
	defer leaktest.AfterTest(t)()
	defer log.Scope(t).Close(t)

	ctx := context.Background()
	engine := NewDefaultInMemForTesting()
	defer engine.Close()

	err := MVCCPut(ctx, engine, nil, testKey1,
		hlc.Timestamp{WallTime: 1}, hlc.ClockTimestamp{}, value1, nil)
	require.NoError(t, err)

	_, _, _, err = MVCCDeleteRange(ctx, engine, nil, testKey1, testKey2,
		0, hlc.Timestamp{}, hlc.ClockTimestamp{}, nil, false)
	require.NoError(t, err)
}

The test currently fails with:

--- FAIL: TestMVCCDeleteRangeInlineEncountersMVCCVersion (0.02s)
    test_log_scope.go:161: test logs captured to: /tmp/_tmp/d1091de82354dd1e885605d51bd315be/logTestMVCCDeleteRangeInlineEncountersMVCCVersion2996635390
    test_log_scope.go:79: use -show-logs to present logs inline
    mvcc_test.go:1849:
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/storage/mvcc_test.go:1849
        	Error:      	Received unexpected error:
        	            	WriteTooOldError: write for key "/db1" at timestamp 0,0 too old; wrote at 0.000000001,1
        	Test:       	TestMVCCDeleteRangeInlineEncountersMVCCVersion

We should stop setting this flag for inline DeleteRange requests. We may also want to scan at MaxTimestamp instead of timestamp 0 so that non-inline values are returned to the mutation phase for the error check, instead of ignored by the search phase.

Jira issue: CRDB-26952