-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvprober: crdb_internal.probe_ranges writes to global keyspace, will corrupt tables #101549
Description
kvprober derives target probe keys from range split points by scanning meta2. It is careful to wrap the split points using keys.RangeProbeKey. This constructs a probe key that is unique (only used by kvprober) and within the local keyspace, ensuring isolation from other workloads.
cockroach/pkg/kv/kvprober/planner.go
Lines 269 to 273 in f13081c
| if rangeDesc.RangeID == 1 { | |
| plans[i].Key = keys.RangeProbeKey(keys.MustAddr(keys.LocalMax)) | |
| } else { | |
| plans[i].Key = keys.RangeProbeKey(rangeDesc.StartKey) | |
| } |
crdb_internal.probe_ranges() is a builtin function that manually performs kvprobes across the cluster. It was intended to work the same way. However, it is missing the keys.RangeProbeKey wrapping over range split points. As a result, it probes directly into the global keyspace.
cockroach/pkg/sql/sem/builtins/generator_probe_ranges.go
Lines 212 to 218 in f13081c
| key := desc.StartKey.AsRawKey() | |
| if desc.RangeID == 1 { | |
| // The first range starts at KeyMin, but the replicated keyspace starts only at keys.LocalMax, | |
| // so there is a special case here. | |
| key = keys.LocalMax | |
| } | |
| return p.rangeProber.RunProbe(ctx, key, p.isWrite) |
cockroach/pkg/sql/rangeprober/range_prober.go
Lines 36 to 41 in f13081c
| op := r.ops.Read | |
| if isWrite { | |
| op = r.ops.Write | |
| } | |
| // NB: intentionally using a separate txn per probe to avoid undesirable cross-probe effects. | |
| return r.db.Txn(ctx, op(key)) |
The consequence of this is that calls to crdb_internal.probe_range will place MVCC deletion tombstones at the start key of each range in the system. In some ranges (tsdb), this corruption will trigger assertion failures later on (#101440). In others (user tables), this will silently corrupt user data.
We need to fix this. When doing so, we should also add guardrails to prevent the kvprober module from touching the global keyspace.
Jira issue: CRDB-27004