Skip to content

ccl/backupccl: TestDataDriven failed #96149

Closed
Closed
ccl/backupccl: TestDataDriven failed#96149
Assignees
stevendanna
Labels
C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.
Milestone
23.1
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Jan 29, 2023

ccl/backupccl.TestDataDriven failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

Fatal error:

panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

Stack:

goroutine 543259385 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3fe8020?, {0x6e7c438, 0xc070408990})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4fccb00, 0xc00790bf00})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0xc04581a000, {0xc01afac4c0, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0x10de
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6e7c438?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0xaa
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:960 +0x1686
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x21b
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc0380cd140, {0x6e7c438, 0xc070408990}, 0xc040349f30)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e7c438, 0xc070408990})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b
Log preceding fatal error 

        ----
        2 newname 2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:125:
        exec-sql [0 args]
        BACKUP TENANT 2 INTO 'nodelocal://1/tenant2'
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:129:
        exec-sql [0 args]
        RESTORE TENANT 2 FROM LATEST IN 'nodelocal://1/tenant2' WITH tenant_name = 'another-name';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:133:
        query-sql [0 args]
        SELECT id,name,service_mode FROM system.tenants WHERE name = 'another-name';
        ----
        3 another-name 2
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
*
* ERROR: a panic has occurred!
* for 3 in test_certs: tenant client cert not found
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213
*   | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36
*   | github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:960
*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-23967

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions