WriteTooOldErrors are handled specially: We try to delay returning this error until the end of the transaction in order to lay down more intents and do more discovery of conflicting timestamps. However, this delay undermines the SQL-level "rewind" functionality which can transparently retry the first statement in a transaction.
We have a strong preference for transparent retries whenever possible. In #22234, we disabled the delayed WriteTooOldError behavior for CPut and Increment, because we know that a transparent RefreshSpans will fail, and if we delay the WriteTooOldError to COMMIT, it will trigger a client-visible retry. This had the effect of ensuring that if the first statement was an INSERT, we'd retry internally, but if it was an UPDATE (including an UPDATE clause in an INSERT ON CONFLICT) we may trigger a retry at COMMIT time.
I think that #22234 didn't go far enough: Nearly all Put operations currently originated by SQL are logically "conditional puts" that are done in separate Get and Put steps. The RefreshSpans is guaranteed to fail just as it is for CPut. We should be returning WriteTooOldError immediately for these regular puts too. Doing this for all Puts effectively disables the "delayed WTOE" behavior, although I think Puts where the delayed WTOE would be beneficial are rare in our current usage.
I also think that #22234 went too far in making WTOE immediate for all CPuts. In the first statement of a transaction, an immediate WTOE maximizes our chance of a transparent retry. But in later statements, CPuts as well as regular Puts benefit from continuing to lay down intents. (This would prevent the O(n^2) behavior from #22234 (comment). In scenarios with high contention on multiple keys, we currently have quadratic behavior for operations using CPut but better performance for operations using Put)
I suggest addressing this in two parts. First, disable the "delayed WTOE" behavior for Puts (and probably just for everything). I think this is probably backportable (although it has the risk of making the aforementioned. O(n^2) behavior more likely for high-contention UPDATEs). Second, introduce a new flag to be plumbed down from SQL to indicate whether we are currently able to autoretry an immediate WTOE, and use this flag instead of the type of request to decide whether WTOEs are immediate or delayed. (I'll only do the first part myself; I need to hand off the latter part to someone else).
WriteTooOldErrorsare handled specially: We try to delay returning this error until the end of the transaction in order to lay down more intents and do more discovery of conflicting timestamps. However, this delay undermines the SQL-level "rewind" functionality which can transparently retry the first statement in a transaction.We have a strong preference for transparent retries whenever possible. In #22234, we disabled the delayed WriteTooOldError behavior for
CPutandIncrement, because we know that a transparentRefreshSpanswill fail, and if we delay the WriteTooOldError to COMMIT, it will trigger a client-visible retry. This had the effect of ensuring that if the first statement was an INSERT, we'd retry internally, but if it was an UPDATE (including an UPDATE clause in an INSERT ON CONFLICT) we may trigger a retry at COMMIT time.I think that #22234 didn't go far enough: Nearly all Put operations currently originated by SQL are logically "conditional puts" that are done in separate Get and Put steps. The RefreshSpans is guaranteed to fail just as it is for CPut. We should be returning WriteTooOldError immediately for these regular puts too. Doing this for all Puts effectively disables the "delayed WTOE" behavior, although I think Puts where the delayed WTOE would be beneficial are rare in our current usage.
I also think that #22234 went too far in making WTOE immediate for all CPuts. In the first statement of a transaction, an immediate WTOE maximizes our chance of a transparent retry. But in later statements, CPuts as well as regular Puts benefit from continuing to lay down intents. (This would prevent the
O(n^2)behavior from #22234 (comment). In scenarios with high contention on multiple keys, we currently have quadratic behavior for operations using CPut but better performance for operations using Put)I suggest addressing this in two parts. First, disable the "delayed WTOE" behavior for Puts (and probably just for everything). I think this is probably backportable (although it has the risk of making the aforementioned.
O(n^2)behavior more likely for high-contention UPDATEs). Second, introduce a new flag to be plumbed down from SQL to indicate whether we are currently able to autoretry an immediate WTOE, and use this flag instead of the type of request to decide whether WTOEs are immediate or delayed. (I'll only do the first part myself; I need to hand off the latter part to someone else).