-
Notifications
You must be signed in to change notification settings - Fork 4.1k
pkg/sql/sqlstats/insights: data race when running TestConsumeJoinToken #85988
Copy link
Copy link
Closed
Labels
A-sql-observabilityRelated to observability of the SQL layerRelated to observability of the SQL layerC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Description
#85350 introduced a data race observed when running the TestConsumeJoinToken test. The test has been failing pretty consistently since then (see TC history).
Stack:
==================
WARNING: DATA RACE
Read at 0x00c001648f68 by goroutine 451:
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).ObserveStatement()
<autogenerated>:1 +0x48
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage.(*Container).RecordStatement()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage/ss_mem_writer.go:178 +0x1608
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).RecordStatement.func1()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:41 +0x100
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).recordStatsOrSendMemoryPressureSignal()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:65 +0x34
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).RecordStatement()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:40 +0xa0
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*StatsCollector).RecordStatement()
<autogenerated>:1 +0xd8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).recordStatementSummary()
github.com/cockroachdb/cockroach/pkg/sql/executor_statement_metrics.go:202 +0x9fc
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1210 +0x1264
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2060
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x6b0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1943 +0x518
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1947 +0xe0c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x2d8
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:212 +0xe4
Previous write at 0x00c001648f68 by goroutine 25:
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.newConcurrentBufferIngester.func2()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:161 +0xbc
github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils.(*ConcurrentBufferGuard).syncLocked()
github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils/concurrent_buffer_guard.go:135 +0x58
github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils.(*ConcurrentBufferGuard).ForceSync()
github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils/concurrent_buffer_guard.go:120 +0x68
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.concurrentBufferIngester.Start.func2()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:80 +0x58
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489 +0x150
Goroutine 451 (running) created at:
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0x838
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:743 +0x8a0
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:540 +0x1e8
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Exec()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:521 +0x10c
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire.func1()
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:157 +0x87c
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1()
github.com/cockroachdb/cockroach/pkg/kv/db.go:963 +0x50
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:960 +0x74
github.com/cockroachdb/cockroach/pkg/kv.runTxn()
github.com/cockroachdb/cockroach/pkg/kv/db.go:962 +0x5c
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
github.com/cockroachdb/cockroach/pkg/kv/db.go:925 +0xc0
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
github.com/cockroachdb/cockroach/pkg/kv/db.go:904 +0x1ec
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire()
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:170 +0x1d4
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func1()
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:492 +0x258
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall()
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x40
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan.func1()
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x68
Goroutine 25 (running) created at:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x488
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0x230
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.concurrentBufferIngester.Start()
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:74 +0x13c
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).Start()
<autogenerated>:1 +0x88
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:528 +0x150
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).Start()
github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:412 +0x238
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1238 +0x1fc
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
github.com/cockroachdb/cockroach/pkg/server/server.go:1565 +0x3404
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
github.com/cockroachdb/cockroach/pkg/server/server.go:964 +0x3c
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
github.com/cockroachdb/cockroach/pkg/server/testserver.go:577 +0x54
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:328 +0x154
github.com/cockroachdb/cockroach/pkg/server.TestConsumeJoinToken()
github.com/cockroachdb/cockroach/pkg/server/addjoin_test.go:38 +0x1dc
testing.tRunner()
GOROOT/src/testing/testing.go:1439 +0x18c
testing.(*T).Run.func1()
GOROOT/src/testing/testing.go:1486 +0x44
Jira issue: CRDB-18514
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-sql-observabilityRelated to observability of the SQL layerRelated to observability of the SQL layerC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.