Skip to content

sql: Panic after moving table from REGIONAL BY ROW to REGIONAL BY TABLE and back to REGIONAL BY ROW #61751

Closed
#61889
Closed
sql: Panic after moving table from REGIONAL BY ROW to REGIONAL BY TABLE and back to REGIONAL BY ROW#61751
#61889
Assignees
arulajmani
Labels
A-multiregionRelated to multi-regionC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
@ajstorm

Description

@ajstorm
ajstorm
opened on Mar 9, 2021

When running the included script, ALTERs from REGIONAL BY ROW to REGIONAL BY TABLE, and back to REGIONAL BY ROW will succeed, but then eventually panic with the below stack. Hit this on ajstorm_zone_config_force_table branch, but it might also be reproducible on master.

zone_configs_repro.scr.zip

* ERROR: a panic has occurred!
* use of enum metadata before hydration as an enum
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	/usr/local/opt/go/libexec/src/runtime/panic.go:969
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql/types.(*T).ensureHydratedEnum
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2415
*   | github.com/cockroachdb/cockroach/pkg/sql/types.(*T).EnumGetIdxOfPhysical
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2381
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.GetEnumComponentsFromPhysicalRep
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3977
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeUntaggedDatum
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:674
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeTableValue
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:524
*   | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:146
*   | github.com/cockroachdb/cockroach/pkg/sql.indexCoveringsForPartitioning
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:202
*   | github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:118
*   | github.com/cockroachdb/cockroach/pkg/sql/catalog.ForEachIndex
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descriptor.go:613
*   | github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:103
*   | github.com/cockroachdb/cockroach/pkg/sql.writeZoneConfig
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:903
*   | github.com/cockroachdb/cockroach/pkg/sql.RemoveIndexZoneConfigs
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:989
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes.func2
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:69
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772
*   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:68
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:66
*   | github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:158
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1178
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1179
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255
*   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
*   | 	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
*   | runtime.goexit
*   | 	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1374
* Wraps: (4) use of enum metadata before hydration as an enum
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
panic: use of enum metadata before hydration as an enum [recovered]
	panic: use of enum metadata before hydration as an enum

goroutine 48849 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000557d80, 0x943d0a0, 0xc0015362c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x8196600, 0xc014f7ac80)
	/usr/local/opt/go/libexec/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/sql/types.(*T).ensureHydratedEnum(...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2415
github.com/cockroachdb/cockroach/pkg/sql/types.(*T).EnumGetIdxOfPhysical(0xc0109f7680, 0xc006d6434e, 0x1, 0x1, 0x1, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/types/types.go:2381 +0x426
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.GetEnumComponentsFromPhysicalRep(0xc0109f7680, 0xc006d6434e, 0x1, 0x1, 0x0, 0x0, 0xc006d6434e, 0x1, 0x1, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3977 +0x4d
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeUntaggedDatum(0xc00768ce10, 0xc0109f7680, 0xc006d6434d, 0x2, 0x2, 0x0, 0x6, 0x0, 0x0, 0x2, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:674 +0x3385
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeTableValue(0xc00768ce10, 0xc0109f7680, 0xc006d6434c, 0x3, 0x3, 0x0, 0x6, 0x0, 0x0, 0x8, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/column_type_encoding.go:524 +0x165
github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodePartitionTuple(0xc00768ce10, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc0012c7288, 0xc0012c73c8, 0xc006d6434c, 0x3, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/partition.go:146 +0x492
github.com/cockroachdb/cockroach/pkg/sql.indexCoveringsForPartitioning(0xc00768ce10, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc0012c7288, 0xc0012c73c8, 0xc00768c8b0, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:202 +0xb3f
github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans.func1(0x959dc60, 0xc0120a5c40, 0x10000, 0x959dc60)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:118 +0x1ae
github.com/cockroachdb/cockroach/pkg/sql/catalog.ForEachIndex(0x95ac880, 0xc0012c7200, 0xc006010000, 0xc00768c9c8, 0xc00768cab0, 0xc00de56800)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descriptor.go:613 +0x11f
github.com/cockroachdb/cockroach/pkg/sql.GenerateSubzoneSpans(0xc0007ed500, 0xff49ba3b6e8caeb5, 0x6f341dfbc6c164a5, 0xc0017be500, 0xc0017be500, 0x0, 0x95ac880, 0xc0012c7200, 0xc005dc7c00, 0x6, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/partition_utils.go:103 +0x375
github.com/cockroachdb/cockroach/pkg/sql.writeZoneConfig(0x943d160, 0xc011f93290, 0xc0036ff830, 0xc000000040, 0x95ac880, 0xc0012c7200, 0xc0109f78c0, 0xc001594000, 0x0, 0x0, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:903 +0x498
github.com/cockroachdb/cockroach/pkg/sql.RemoveIndexZoneConfigs(0x943d160, 0xc011f93290, 0xc0036ff830, 0xc001594000, 0x40, 0xc00768d670, 0x1, 0x1, 0x0, 0xc0007ed500)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:989 +0x2d5
github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes.func2(0x943d160, 0xc011f93290, 0xc0036ff830, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:69 +0x111
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1(0x943d160, 0xc011f93290, 0xc0036ff830, 0x3ff0000000000000, 0x9591000)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772 +0x43
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc0036ff830, 0x943d160, 0xc011f93290, 0xc00768dbe0, 0xc0036ff830, 0x7ebfc00)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828 +0xdc
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc0010ceb60, 0x943d160, 0xc011f93290, 0xc00768dee0, 0xc00781c900, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771 +0x105
github.com/cockroachdb/cockroach/pkg/sql/gcjob.gcIndexes(0x943d160, 0xc011f93290, 0xc001594000, 0x40, 0xc0174b5d00, 0xc0100ff550, 0x76170c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/index_garbage_collection.go:68 +0x31a
github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC(0x943d160, 0xc011f93290, 0xc001594000, 0xc01889ecc0, 0xc0174b5d00, 0xc00a3a6740, 0x28)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:66 +0x225
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume(0x8e118e573060001, 0x943d160, 0xc011f93290, 0x82a5600, 0xc004c84ee0, 0x0, 0x0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:158 +0x578
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc00159e0c0, 0x93e7720, 0xc00ac46af8, 0x943d160, 0xc011f93290, 0x82a5600, 0xc004c84ee0, 0xc00768fa50)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1178 +0xae
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc000789040, 0x943d160, 0xc011f93230, 0x82a5600, 0xc004c84ee0, 0x93e7720, 0xc00ac46af8, 0xc0126f72c0, 0xc002526901, 0x7, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1179 +0x885
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc000789040, 0x943d0a0, 0xc018c879c0, 0x93e7720, 0xc00ac46af8, 0xc0126f72c0, 0xc002526901, 0x7, 0xc00d7be540, 0x16, ...)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x943d0a0, 0xc0015362c0)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000557d80, 0x943d0a0, 0xc0015362c0, 0x0, 0xc012fafb30)
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/storm/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc```

Metadata

Metadata

Assignees

Labels

A-multiregionRelated to multi-regionC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions