-
Notifications
You must be signed in to change notification settings - Fork 4.1k
server: race condition between metric recorder and job scheduler initialization #69522
Copy link
Copy link
Closed
Labels
A-disaster-recoveryA-jobsC-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.T-disaster-recoverybranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.
Description
server.TestPlainHTTPServer failed with artifacts on master @ 392bd80db56a041099afe674c4436e0c8e3c02e6:
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x161
Goroutine 137 (running) created at:
google.golang.org/grpc.(*Server).serveStreams.func1()
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:939 +0x354
google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:530 +0x1a70
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:575 +0x4fc
google.golang.org/grpc.(*Server).serveStreams()
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:925 +0x233
google.golang.org/grpc.(*Server).handleRawConn.func1()
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:875 +0x50
Goroutine 975 (running) created at:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:334 +0x14d
github.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon()
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/job_scheduler.go:347 +0x45
github.com/cockroachdb/cockroach/pkg/jobs.StartJobSchedulerDaemon()
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/job_scheduler.go:469 +0xe5
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1086 +0x12b5
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1855 +0x48c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask()
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:307 +0x129
github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus()
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:788 +0xf6
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus()
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:751 +0xfb
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1728 +0x3afe
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1200 +0x50
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:466 +0x6e
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:278 +0xfe
github.com/cockroachdb/cockroach/pkg/server.TestPlainHTTPServer()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server_test.go:208 +0x173
testing.tRunner()
/usr/local/go/src/testing/testing.go:1193 +0x202
==================
server_test.go:245: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestPlainHTTPServer398611769
testing.go:1092: race detected during execution of test
--- FAIL: TestPlainHTTPServer (1.99s)
Reproduce
To reproduce, try:
make stressrace TESTS=TestPlainHTTPServer PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Parameters in this failure:
- GOFLAGS=-race -parallel=4
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-disaster-recoveryA-jobsC-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.T-disaster-recoverybranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.