Skip to content

server: race condition between metric recorder and job scheduler initialization #69522

@cockroach-teamcity

Description

@cockroach-teamcity

server.TestPlainHTTPServer failed with artifacts on master @ 392bd80db56a041099afe674c4436e0c8e3c02e6:

      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x161

Goroutine 137 (running) created at:
  google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:939 +0x354
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:530 +0x1a70
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:575 +0x4fc
  google.golang.org/grpc.(*Server).serveStreams()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:925 +0x233
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:875 +0x50

Goroutine 975 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28d
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:334 +0x14d
  github.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/job_scheduler.go:347 +0x45
  github.com/cockroachdb/cockroach/pkg/jobs.StartJobSchedulerDaemon()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/job_scheduler.go:469 +0xe5
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1086 +0x12b5
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1855 +0x48c8
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:307 +0x129
  github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:788 +0xf6
  github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:751 +0xfb
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1728 +0x3afe
  github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1200 +0x50
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:466 +0x6e
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:278 +0xfe
  github.com/cockroachdb/cockroach/pkg/server.TestPlainHTTPServer()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server_test.go:208 +0x173
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1193 +0x202
==================
    server_test.go:245: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestPlainHTTPServer398611769
    testing.go:1092: race detected during execution of test
--- FAIL: TestPlainHTTPServer (1.99s)
Reproduce

To reproduce, try:

make stressrace TESTS=TestPlainHTTPServer PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-race -parallel=4

/cc @cockroachdb/server mberhault

This test on roachdash | Improve this report!

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions