kvserver: use non-cancelable contexts in raft application #75656
Description
Describe the problem
As of d064059, raft application on the proposal happens under the proposer's context, which may well be canceled before or mid-way through the application. However, raft application should not care about the cancellation, and we just diagnosed a bug where it did.
Also, we had problems with cancellation on the raft goroutine in the past and check against it, but evidently not well enough yet.
Expected behavior
The raft apply loop should use a ctx that is not cancelable. It makes sense to derive certain parts from the proposer - such as a trace span - but not the cancellation policy. That is, in
cockroach/pkg/kv/kvserver/replica_application_decoder.go
Lines 139 to 171 in 852a80c
we should not use a child context of cmd.proposal.ctx but create one derived from ctx (the worker ctx) or context.Background().
Additional data / screenshots
Should also write a test that uses a TestingPostApplyHook to ensure that the ctx is not cancelable, or at least that the cancellation doesn't carry over to the child. (Or more simply, that a value stashed in the proposer context is not visible at apply time).
@erikgrinaker didn't we put in some invariant check about this sometime recently?
Jira issue: CRDB-12769