sql: prepared statement with AS OF SYSTEM TIME can create txn atomicity violation #68216
Description
In 078c58a, we added an assertion that transaction timestamps are not manually set using the (*Txn).SetFixedTimestamp API after a transaction has already been used. This would be very unsafe, as it would allow transactions to silently change timestamps and violate atomicity. It turns out that we do hit this assertion, which revealed that something was wrong.
The good news is that even without the assertion, we don't seem to be silently violating atomicity in these cases, as another assertion in KV notices the changed txn timestamp. The bad news is that this is possible to hit thanks to the unchecked use of SetFixedTimestamp during statement preparation.
> create table t (x int);
CREATE TABLE
> insert into t values (1);
INSERT 1
> select crdb_internal_mvcc_timestamp, * from t;
crdb_internal_mvcc_timestamp | x
---------------------------------+----
1627526754922910000.0000000000 | 1
(1 row)
> begin as of system time (1627526754922910000.0000000000-1)::string;
BEGIN
OPEN> select * from t;
x
-----
(0 rows)
OPEN> prepare y as select * from t as of system time 1627526754922910000.0000000000;
PREPARE
OPEN> select * from t;
ERROR: internal error: unexpected batch read timestamp: 1627526754.922910000,0. Expected refreshed timestamp: 1627526754.922909999,0. ba: Scan [/Table/59/1,/Table/59/2), [txn: ad8d53e5]. txn: "sql txn" meta={id=ad8d53e5 pri=0.03468194 epo=0 ts=1627526754.922910000,0 min=1627526754.922909999,0 seq=0} lock=false stat=PENDING rts=1627526754.922910000,0 wto=false gul=1627526754.922910000,0
SQLSTATE: XX000
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:160: SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:285: SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105: SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:240: SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:513: Send()
github.com/cockroachdb/cockroach/pkg/kv/db.go:831: sendUsingSender()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:982: Send()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:224: func1()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:443: fetch()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:589: nextBatch()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:147: NextKV()
github.com/cockroachdb/cockroach/pkg/sql/colfetcher/cfetcher.go:839: nextBatch()
github.com/cockroachdb/cockroach/pkg/sql/colfetcher/cfetcher.go:798: nextAdapter()
github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:299: PerformOperation()
github.com/cockroachdb/cockroach/pkg/sql/colfetcher/cfetcher.go:790: NextBatch()
github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_scan.go:97: Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:59: Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:125: Next()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:239: nextAdapter()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91: CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:243: next()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282: Run()
github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:257: Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:525: Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1262: PlanAndRun()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1110: execWithDistSQLEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:972: dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:690: execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:126: execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1625: func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1627: execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1549: run()
Jira issue: CRDB-8932