Gateway flows will sometimes use the Root txn erroneously for evaluating transactions, when the should be using a Leaf.

The processes of creating the processors/operators in a flow makes a copy of the EvalCtx from the FlowCtx by using NewEvalCtx. This generally happens when processors/operators are instantiated. The EvalCtx contains a *client.Txn. On the gateway, that transaction is initially the Root transaction. However, if there's any remote flows or if the gateway flow has any concurrency, we'll later switch the transaction that the gateway flow uses to a Leaf. We're switching FlowCtx.Txn and FlowCtx.EvalCtx.Txn. Unfortunately,
Unfortunately, by the time we switch that txn, all the processors have already captured the old one for the purposes of expression evaluation. This is bad because the gateway flow can have concurrency in it, and RootTxns don't support concurrency (#25329). Worse, RootTxn cannot be used in conjunction with remote leaves (this is another type of concurrency, really) because the Root might refresh at an inopportune time causing write skew.

The txn is used during expression evaluation by some built-in function - many pg_ ones use it to query system tables using the internalExecutor.

I think the solution here is to pass the transaction explicitly to the builtins that need it (one way or another). In fact I think it'd be a good idea to take the txn out of the EvalCtx completely and leave the EvalCtx for immutable session attributes.
This is related to #15670 which complains that the context.Context that captured through the EvalCtx and used by expression evaluation is similarly the wrong one.
It is also related to #41222, which complains that we might be missing to collect the metadata of transactions used for expression evaluation. This all speaks to the fact that expression evaluation was kinda bolted onto DistSQL processors without sufficient smoothing.

Jira issue: CRDB-5401