Skip to content

pkg/sql/sqlstats/insights: data race when running TestConsumeJoinToken #85988

@renatolabs

Description

@renatolabs

#85350 introduced a data race observed when running the TestConsumeJoinToken test. The test has been failing pretty consistently since then (see TC history).

Stack:

==================
WARNING: DATA RACE
Read at 0x00c001648f68 by goroutine 451:
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).ObserveStatement()
      <autogenerated>:1 +0x48
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage.(*Container).RecordStatement()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage/ss_mem_writer.go:178 +0x1608
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).RecordStatement.func1()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:41 +0x100
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).recordStatsOrSendMemoryPressureSignal()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:65 +0x34
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*ApplicationStats).RecordStatement()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/appStats.go:40 +0xa0
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*StatsCollector).RecordStatement()
      <autogenerated>:1 +0xd8
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).recordStatementSummary()
      github.com/cockroachdb/cockroach/pkg/sql/executor_statement_metrics.go:202 +0x9fc
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1210 +0x1264
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2060
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x6b0
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1943 +0x518
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1947 +0xe0c
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x2d8
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:212 +0xe4

Previous write at 0x00c001648f68 by goroutine 25:
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.newConcurrentBufferIngester.func2()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:161 +0xbc
  github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils.(*ConcurrentBufferGuard).syncLocked()
      github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils/concurrent_buffer_guard.go:135 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils.(*ConcurrentBufferGuard).ForceSync()
      github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils/concurrent_buffer_guard.go:120 +0x68
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.concurrentBufferIngester.Start.func2()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:80 +0x58
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489 +0x150

Goroutine 451 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0x838
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:743 +0x8a0
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:540 +0x1e8
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Exec()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:521 +0x10c
  github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire.func1()
      github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:157 +0x87c
  github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:963 +0x50
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
      github.com/cockroachdb/cockroach/pkg/kv/txn.go:960 +0x74
  github.com/cockroachdb/cockroach/pkg/kv.runTxn()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:962 +0x5c
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:925 +0xc0
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:904 +0x1ec
  github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire()
      github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:170 +0x1d4
  github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func1()
      github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:492 +0x258
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x40
  github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan.func1()
      github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x68

Goroutine 25 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x488
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0x230
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.concurrentBufferIngester.Start()
      github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights/pkg/sql/sqlstats/insights/ingester.go:74 +0x13c
  github.com/cockroachdb/cockroach/pkg/sql/sqlstats/insights.(*concurrentBufferIngester).Start()
      <autogenerated>:1 +0x88
  github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:528 +0x150
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).Start()
      github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:412 +0x238
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1238 +0x1fc
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1565 +0x3404
  github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
      github.com/cockroachdb/cockroach/pkg/server/server.go:964 +0x3c
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:577 +0x54
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:328 +0x154
  github.com/cockroachdb/cockroach/pkg/server.TestConsumeJoinToken()
      github.com/cockroachdb/cockroach/pkg/server/addjoin_test.go:38 +0x1dc
  testing.tRunner()
      GOROOT/src/testing/testing.go:1439 +0x18c
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1486 +0x44

Jira issue: CRDB-18514

Metadata

Metadata

Assignees

Labels

A-sql-observabilityRelated to observability of the SQL layerC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions