Skip to content

server: TestStartupInjectedFailureSingleNode failed [session based leasing issues] #118520

@cockroach-teamcity

Description

@cockroach-teamcity

server.TestStartupInjectedFailureSingleNode failed on master @ 03525787deade5395bac0a8b5735fa7e715e22ef:

        	            	  |   | 	src/runtime/asm_amd64.s:1650
        	            	  | Wraps: (2) list-tenants
        	            	  | Wraps: (3) unexpected value: raw_bytes:"\372N\360\267\n3\002" timestamp:<wall_time:1706680561085812090 > 
        	            	  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
        	            	Wraps: (4) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
        	            	  | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server.go:2187
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
        	            	  | 	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
        	            	  | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1595
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (5) list-tenants
        	            	Wraps: (6) unexpected value: raw_bytes:"\372N\360\267\n3\002" timestamp:<wall_time:1706680561085812090 > 
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
        	Test:       	TestStartupInjectedFailureSingleNode
        	Messages:   	failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode1240926317
--- FAIL: TestStartupInjectedFailureSingleNode (3.54s)

Parameters:

  • attempt=1
  • run=10
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-35766

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.P-0Issues/test failures with a fix SLA of 2 weeksT-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)branch-masterFailures and bugs on the master branch.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions