This commit introduces a new KV request type `ProbeRequest`. This
request performs a mutationless round-trip through the replication
layer. The proximate reason to add this request is cockroachdb#33007, where we want
to fail-fast requests on a Replica until a `ProbeRequest` succeeds.

The alternative to introducing a new request type is using a a `Put` on
`keys.RangeProbeKey` (a key introduced for [kvprober]). However, when
the Replica circuit breakers perform a probe, they have to make sure
that this probe itself doesn't get caught up in the circuit breakers. In
particular, the circuit breaker's request needs special casing with
respect to lease requests (which would otherwise also bounce on the
circuit breaker). But even with that added (not too large a change),
the lease would be required, so we wouldn't be able to apply the
circuit breakers anywhere but the leaseholder (i.e. we would have
to heal the breaker when the lease is "not obtainable". What we are
after is really a per-Replica concept that is disjoint from the lease,
where we check that the replica can manage to get a command into the
log and observe it coming out of the replication layer.

`ProbeRequest` bypasses lease checks and can thus be processed by any
Replica. Upon application, they are guaranteed to result in a forced
error, and we suppress this error above the replication layer (i.e. at
the waiting request). This means that receiving a nil error from the
probe implies that the probe successfully made it through latching
(where it declares a read at its timestamp on the probed key),
evaluation, and application.

While no concrete plans exist for adoption of `ProbeRequest` elsewhere,
this might be of interest for [kvprober] and as a general building block
for quickly assessing overall KV health; for example we may extend cockroachdb#72732
to facilitate sending a probe to the leaseholders of all Ranges in the
system.

`ProbeRequest` is a point request. A ranged version is possible but
since `DistSender` will not return partial responses, it is not
introduced at this point. This can be done when a concrete need
arises.

`ProbeRequest` by default will be routed to the leaseholder, but it
can also be used with the `NEAREST` routing policy.

Some details helpful for reviewers follow.

- to enable testing of this new request type, `TestingApplyFilter` was
  augmented with a `TestingApplyForcedErrFilter` which gets to inspect
  and mutate forced errors below raft. The semantics of
  `TestingApplyFitler` remain unchainged; the newly added field
  `ForcedErr` is always nil when handed to such a filter.
- `ProbeRequest` is the first request type that skips the lease check
  and is not itself a lease check. This prompted a small refactor to
  `GetPrevLeaseForLeaseRequest`, which can now return false when the
  underlying request is a `Probe` and not `{Transfer,Request}Lease`.

[kvprober]: https://github.com/cockroachdb/cockroach/tree/2ad2bee257e78970ce2c457ddd6996099ed6727a/pkg/kv/kvprober
[routing policy]: https://github.com/cockroachdb/cockroach/blob/e1c7d6a55ed147aeb20d6d5c564d3d9bf73d2624/pkg/roachpb/api.proto#L48-L60

Release note: None

This commit introduces a new KV request type `ProbeRequest`. This
request performs a mutationless round-trip through the replication
layer. The proximate reason to add this request is cockroachdb#33007, where we want
to fail-fast requests on a Replica until a `ProbeRequest` succeeds.

The alternative to introducing a new request type is using a `Put` on
`keys.RangeProbeKey` (a key introduced for [kvprober]). However, when
the Replica circuit breakers perform a probe, they have to make sure
that this probe itself doesn't get caught up in the circuit breakers. In
particular, the circuit breaker's request needs special casing with
respect to lease requests (which would otherwise also bounce on the
circuit breaker). But even with that added (not too large a change),
the lease would be required, so we wouldn't be able to apply the
circuit breakers anywhere but the leaseholder (i.e. we would have
to heal the breaker when the lease is "not obtainable". What we are
after is really a per-Replica concept that is disjoint from the lease,
where we check that the replica can manage to get a command into the
log and observe it coming out of the replication layer.

`ProbeRequest` bypasses lease checks and can thus be processed by any
Replica. Upon application, they are guaranteed to result in a forced
error, and we suppress this error above the replication layer (i.e. at
the waiting request). This means that receiving a nil error from the
probe implies that the probe successfully made it through latching
(where it declares a read at its timestamp on the probed key),
evaluation, and application.

While no concrete plans exist for adoption of `ProbeRequest` elsewhere,
this might be of interest for [kvprober] and as a general building block
for quickly assessing overall KV health; for example we may extend cockroachdb#72732
to facilitate sending a probe to the leaseholders of all Ranges in the
system.

`ProbeRequest` is a point request. A ranged version is possible but
since `DistSender` will not return partial responses, it is not
introduced at this point. This can be done when a concrete need
arises.

`ProbeRequest` by default will be routed to the leaseholder, but it
can also be used with the `NEAREST` routing policy.

Some details helpful for reviewers follow.

- to enable testing of this new request type, `TestingApplyFilter` was
  augmented with a `TestingApplyForcedErrFilter` which gets to inspect
  and mutate forced errors below raft. The semantics of
  `TestingApplyFilter` remain unchainged; the newly added field
  `ForcedErr` is always nil when handed to such a filter.
- `ProbeRequest` is the first request type that skips the lease check
  and is not itself a lease check. This prompted a small refactor to
  `GetPrevLeaseForLeaseRequest`, which can now return false when the
  underlying request is a `Probe` and not `{Transfer,Request}Lease`.

[kvprober]: https://github.com/cockroachdb/cockroach/tree/2ad2bee257e78970ce2c457ddd6996099ed6727a/pkg/kv/kvprober
[routing policy]: https://github.com/cockroachdb/cockroach/blob/e1c7d6a55ed147aeb20d6d5c564d3d9bf73d2624/pkg/roachpb/api.proto#L48-L60

Release note: None

This commit introduces a new KV request type `ProbeRequest`. This
request performs a mutationless round-trip through the replication
layer. The proximate reason to add this request is cockroachdb#33007, where we want
to fail-fast requests on a Replica until a `ProbeRequest` succeeds.

The alternative to introducing a new request type is using a `Put` on
`keys.RangeProbeKey` (a key introduced for [kvprober]). However, when
the Replica circuit breakers perform a probe, they have to make sure
that this probe itself doesn't get caught up in the circuit breakers. In
particular, the circuit breaker's request needs special casing with
respect to lease requests (which would otherwise also bounce on the
circuit breaker). But even with that added (not too large a change),
the lease would be required, so we wouldn't be able to apply the
circuit breakers anywhere but the leaseholder (i.e. we would have
to heal the breaker when the lease is "not obtainable". What we are
after is really a per-Replica concept that is disjoint from the lease,
where we check that the replica can manage to get a command into the
log and observe it coming out of the replication layer.

`ProbeRequest` bypasses lease checks and can thus be processed by any
Replica. Upon application, they are guaranteed to result in a forced
error, and we suppress this error above the replication layer (i.e. at
the waiting request). This means that receiving a nil error from the
probe implies that the probe successfully made it through latching
(where it doesn't declare any key, i.e. should not conflict with long-
evaluating requests), evaluation, and application.

While no concrete plans exist for adoption of `ProbeRequest` elsewhere,
this might be of interest for [kvprober] and as a general building block
for quickly assessing overall KV health; for example we may extend cockroachdb#72732
to facilitate sending a probe to the leaseholders of all Ranges in the
system.

`ProbeRequest` is a point request. A ranged version is possible but
since `DistSender` will not return partial responses, it is not
introduced at this point. This can be done when a concrete need
arises.

`ProbeRequest` by default will be routed to the leaseholder, but it
can also be used with the `NEAREST` routing policy.

Some details helpful for reviewers follow.

- to enable testing of this new request type, `TestingApplyFilter` was
  augmented with a `TestingApplyForcedErrFilter` which gets to inspect
  and mutate forced errors below raft. The semantics of
  `TestingApplyFilter` remain unchainged; the newly added field
  `ForcedErr` is always nil when handed to such a filter.
- `ProbeRequest` is the first request type that skips the lease check
  and is not itself a lease check. This prompted a small refactor to
  `GetPrevLeaseForLeaseRequest`, which can now return false when the
  underlying request is a `Probe` and not `{Transfer,Request}Lease`.

[kvprober]: https://github.com/cockroachdb/cockroach/tree/2ad2bee257e78970ce2c457ddd6996099ed6727a/pkg/kv/kvprober
[routing policy]: https://github.com/cockroachdb/cockroach/blob/e1c7d6a55ed147aeb20d6d5c564d3d9bf73d2624/pkg/roachpb/api.proto#L48-L60

Release note: None

72972: kvserver: introduce ProbeRequest r=erikgrinaker a=tbg

This commit introduces a new KV request type `ProbeRequest`. This
request performs a mutationless round-trip through the replication
layer. The proximate reason to add this request is #33007, where we want
to fail-fast requests on a Replica until a `ProbeRequest` succeeds.

The alternative to introducing a new request type is using a a `Put` on
`keys.RangeProbeKey` (a key introduced for [kvprober]). However, when
the Replica circuit breakers perform a probe, they have to make sure
that this probe doesn't get caught up in the circuit breakers
themselves. In particular, the circuit breaker's request needs special
casing with respect to lease requests (which would otherwise also bounce
on the circuit breaker), and in particular we would also like the probe
to be proposed (via the local raft instance) when the local replica is
not the leaseholder, as this allows the circuit breakers to cover more
ground and ascertain that the Replica has access to the replication
layer regardless of its status.

`ProbeRequest` bypasses lease checks and can thus be processed by any
Replica. Upon application, they are guaranteed to result in a forced
error, and we suppress this error above the replication layer (i.e. at
the waiting request). This means that receiving a nil error from the
probe implies that the probe successfully made it through latching,
evaluation, and application.

While no concrete plans exist for adoption of `ProbeRequest` elsewhere,
this might be of interest for [kvprober] and as a general building block
for quickly assessing overall KV health; for example we may extend #72732
to facilitate sending a probe to the leaseholders of all Ranges in the
system.

`ProbeRequest` is a point request. A ranged version is possible but
since `DistSender` will not return partial responses, it is not
introduced at this point. This can be done when a concrete need
arises.

`ProbeRequest` by default will be routed to the leaseholder, but it
can also be used with the `NEAREST` routing policy.

[kvprober]: https://github.com/cockroachdb/cockroach/tree/2ad2bee257e78970ce2c457ddd6996099ed6727a/pkg/kv/kvprober
[routing policy]: https://github.com/cockroachdb/cockroach/blob/e1c7d6a55ed147aeb20d6d5c564d3d9bf73d2624/pkg/roachpb/api.proto#L48-L60

Release note: None


Co-authored-by: Tobias Grieger <tobias.schottdorf@gmail.com>
Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>

This commit introduces a new KV request type `ProbeRequest`. This
request performs a mutationless round-trip through the replication
layer. The proximate reason to add this request is cockroachdb#33007, where we want
to fail-fast requests on a Replica until a `ProbeRequest` succeeds.

The alternative to introducing a new request type is using a `Put` on
`keys.RangeProbeKey` (a key introduced for [kvprober]). However, when
the Replica circuit breakers perform a probe, they have to make sure
that this probe itself doesn't get caught up in the circuit breakers. In
particular, the circuit breaker's request needs special casing with
respect to lease requests (which would otherwise also bounce on the
circuit breaker). But even with that added (not too large a change),
the lease would be required, so we wouldn't be able to apply the
circuit breakers anywhere but the leaseholder (i.e. we would have
to heal the breaker when the lease is "not obtainable". What we are
after is really a per-Replica concept that is disjoint from the lease,
where we check that the replica can manage to get a command into the
log and observe it coming out of the replication layer.

`ProbeRequest` bypasses lease checks and can thus be processed by any
Replica. Upon application, they are guaranteed to result in a forced
error, and we suppress this error above the replication layer (i.e. at
the waiting request). This means that receiving a nil error from the
probe implies that the probe successfully made it through latching
(where it doesn't declare any key, i.e. should not conflict with long-
evaluating requests), evaluation, and application.

While no concrete plans exist for adoption of `ProbeRequest` elsewhere,
this might be of interest for [kvprober] and as a general building block
for quickly assessing overall KV health; for example we may extend cockroachdb#72732
to facilitate sending a probe to the leaseholders of all Ranges in the
system.

`ProbeRequest` is a point request. A ranged version is possible but
since `DistSender` will not return partial responses, it is not
introduced at this point. This can be done when a concrete need
arises.

`ProbeRequest` by default will be routed to the leaseholder, but it
can also be used with the `NEAREST` routing policy.

Some details helpful for reviewers follow.

- to enable testing of this new request type, `TestingApplyFilter` was
  augmented with a `TestingApplyForcedErrFilter` which gets to inspect
  and mutate forced errors below raft. The semantics of
  `TestingApplyFilter` remain unchainged; the newly added field
  `ForcedErr` is always nil when handed to such a filter.
- `ProbeRequest` is the first request type that skips the lease check
  and is not itself a lease check. This prompted a small refactor to
  `GetPrevLeaseForLeaseRequest`, which can now return false when the
  underlying request is a `Probe` and not `{Transfer,Request}Lease`.

[kvprober]: https://github.com/cockroachdb/cockroach/tree/2ad2bee257e78970ce2c457ddd6996099ed6727a/pkg/kv/kvprober
[routing policy]: https://github.com/cockroachdb/cockroach/blob/e1c7d6a55ed147aeb20d6d5c564d3d9bf73d2624/pkg/roachpb/api.proto#L48-L60

Release note: None