c2c/admission: hit observed raft log position less than per-stream lower bound panic during race run of TestTenantStreamingMultipleNodes #106589
Description
UPDATE: I have repro'd the failure on a cleaner branch here. See newer CI failure here.
ORIGINAL MESSAGE: See the CI failure here in a draft PR that attempts to unskip TestTenantStreamingMultiNodes under stress race. Full test log for this run here. I have seen this stack trace on master Extended CI, but have yet to reproduce this locally on my gce worker.
Full stack trace:
[TestTenantStreamingMultipleNodes] === RUN TestTenantStreamingMultipleNodes
test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1823897493
test_log_scope.go:81: use -show-logs to present logs inline
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 observed raft log position less than per-stream lower bound (log-position=0/0 <= log-position=9/6600)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !goroutine 255859 [running]:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !runtime/debug.Stack()
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! GOROOT/src/runtime/debug/stack.go:24 +0x72
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc010fa16e0, {{{0xc010f040c0, 0x24}, {0x94dc09b, 0x1}, {0x94dc09a, 0x1}, {0x94dc09b, 0x1}}, 0x1770ab40ea3045ae, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/clog.go:261 +0xb7
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xd499400, 0xc011b828a0}, 0x2, 0x4, 0x0, 0x0, {0x9603daa, 0x46}, {0xc039c01720, 0x2, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x6e6
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf({0xd499400, 0xc011b828a0}, {0x9603daa, 0x46}, {0xc039c01720, 0x2, 0x2})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:848 +0x9d
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowtokentracker.(*Tracker).Track(0xc04b7bade0, {0xd499400, 0xc011b828a0}, 0x0, 0xbd, {0xc0154167a0?, 0x4?})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 +0x85b
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle.(*Handle).deductTokensForInner(0xc02ddcfc00, {0xd499400, 0xc011b828a0}, 0x0?, {0x1350ce80?, 0xc0fb967b30?}, 0xbd)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle/kvflowhandle.go:162 +0x394
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle.(*Handle).DeductTokensFor(0xc02ddcfc00, {0xd499400, 0xc011b828a0}, 0x7d?, {0x2c?, 0xc015416880?}, 0x1?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle/kvflowhandle.go:141 +0x72
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.maybeDeductFlowTokens({0xd499400, 0xc0f155cf60}, {0xd4df900, 0xc02ddcfc00}, {0xc05ce6e2d0, 0x3, 0x3}, {0xc037f647e0, 0x3, 0x3})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:1051 +0x2e5
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.proposeBatch({0xd499400, 0xc0f155cf60}, {0x7fcccd588ca0, 0xc0efe8a880}, {0xd4b9258, 0xc11e1c02d0}, {0xc037f647e0, 0x3, 0x3}, {0xc05ce6e2d0, ...}, ...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:1028 +0x24f
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc0efe8b0d0, {0xd499400, 0xc0f155cf60}, {0xd4b9258, 0xc11e1c02d0})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:621 +0x1c97
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x0?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:804 +0xfd
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1(...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:2042
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc0efe8a880, 0x1, 0xc015418760)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:2043 +0x3ae
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:801 +0x3a5
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757 +0x210
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc00396aa80, 0xd499400?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:660 +0x1d0
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc007028e70, {0xd499400, 0xc006dd95f0}, {0xd47cc40, 0xc00396aa80}, 0xc004303500)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x2f6
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xd499400, 0xc006dd95f0})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x9b
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f7
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x61a
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !For more context, check log files in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1823897493
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !****************************************************************************
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !This node experienced a fatal error (printed above), and as a result the
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !process is terminating.
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !will try to determine the root cause, recommend next steps, and we can
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !improve CockroachDB based on your report.
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !Please submit a crash report by following the instructions here:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! https://github.com/cockroachdb/cockroach/issues/new/choose
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !If you would rather not post publicly, please contact us directly at:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 ! support@cockroachlabs.com
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !The Cockroach Labs team appreciates your feedback.
Jira issue: CRDB-29634