datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m0.002687657s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m10.002882598s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m20.003105345s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m30.003650071s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m40.003936266s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 1m50.003992222s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m0.004422616s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m10.005460163s
    datadriven.go:352: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/schemachangerccl/sctestbackupccl/sctestbackupccl_test_/sctestbackupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/schemachangerccl/testdata/end_to_end/alter_table_alter_primary_key_rbr/alter_table_alter_primary_key_rbr.definition:58: still running after 2m20.005641946s
    panic.go:635: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr3731241393
--- FAIL: TestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr (149.86s)
=== RUN   TestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_12_of_24
--- FAIL: TestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_12_of_24 (97.82s)
=== RUN   TestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_12_of_24/restore_database
    sql_runner.go:117: 
        	Error Trace:	pkg/testutils/sqlutils/sql_runner.go:130
        	            				pkg/testutils/sqlutils/sql_runner.go:391
        	            				pkg/sql/schemachanger/sctest/framework.go:968
        	            				pkg/sql/schemachanger/sctest/backup.go:587
        	Error:      	Received unexpected error:
        	            	query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
        	            	  | 	pkg/testutils/sqlutils/sql_runner.go:394
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
        	            	  | 	pkg/testutils/soon.go:71
        	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
        	            	  | 	pkg/util/retry/retry.go:346
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
        	            	  | 	pkg/testutils/soon.go:77
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
        	            	  | 	pkg/testutils/sqlutils/sql_runner.go:130
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
        	            	  | 	pkg/testutils/sqlutils/sql_runner.go:391
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.waitForSchemaChangesToFinish
        	            	  | 	pkg/sql/schemachanger/sctest/framework.go:968
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.exerciseBackupRestore.func1
        	            	  | 	pkg/sql/schemachanger/sctest/backup.go:587
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1690
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1700
        	            	Wraps: (2) query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	  |
        	            	  | got:
        	            	  | running, NEW SCHEMA CHANGE, ALTER TABLE multiregion_db.public.table_regional_by_row ADD … NG HASH
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
--- FAIL: TestBackupSuccess_multiregion_alter_table_alter_primary_key_rbr/post_commit_stage_12_of_24/restore_database (46.60s)