Skip to content

sql: support version numbers on descriptor validation#76166

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:validationVersion
Feb 8, 2022
Merged

sql: support version numbers on descriptor validation#76166
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:validationVersion

Conversation

@fqazi
Copy link
Copy Markdown
Collaborator

@fqazi fqazi commented Feb 7, 2022

Previously, the descriptor validation code did not
take a version number, so it was not possible to version
gate new validation logic. This was inadequate because
when new fields are introduced we don't want their validation
to kick in for certain cases like the debug doctor, such as any
new fields with non-zero defaults. To address this, this patch add supports
for version numbers inside the validation, and updates unit tests
to pass this in as well. It also adds a new option on debug doctor
to run a version of validation.

Release note (cli change): Add new optional version argument
to the doctor examine command. This can be used to enable /
disable validation when examining older zip directories.

@fqazi fqazi requested review from a team, ajwerner and postamar February 7, 2022 16:15
@fqazi fqazi requested review from a team as code owners February 7, 2022 16:15
@fqazi fqazi requested review from dt and removed request for a team February 7, 2022 16:15
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@fqazi fqazi force-pushed the validationVersion branch from fc97740 to 1996708 Compare February 7, 2022 16:58
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 54 of 146 files at r1.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @dt, @fqazi, and @postamar)


pkg/sql/catalog/descs/descriptor.go, line 297 at r1 (raw file):

	descs, err = tc.withReadFromStore(mutable, func() ([]catalog.MutableDescriptor, error) {
		uncommittedDB, _ := tc.uncommitted.getByID(parentID).(catalog.DatabaseDescriptor)
		version := tc.settings.Version.ActiveVersion(ctx)

shouldn't this use the version we already initialized and stored in the collection?


pkg/sql/catalog/desctestutils/descriptor_test_utils.go, line 35 at r1 (raw file):

// TestingGetDatabaseDescriptor retrieves a database descriptor directly from
// the kv layer.
func TestingGetDatabaseDescriptor(

Having reviewed a lot of this change, I wonder to what extent this plumbing here is making things a lot more painful. Should we make a variant which takes a version and just hard-code it to the latest version in the existing variant and plumb that through to the new variant? It feels like it would reduce the number of changes dramatically.

fqazi added a commit to fqazi/cockroach that referenced this pull request Feb 7, 2022
Previously, the descriptor validation code did not
take a version number, so it was not possible to version
gate new validation logic. This was inadequate because
when new fields are introduced we don't want their validation
to kick in for certain cases like the debug doctor, such as any
new fields with non-zero defaults. To address this, this patch add supports
for version numbers inside the validation, and updates unit tests
to pass this in as well. It also adds a new option on debug doctor
to run a version of validation.

Release note (cli change): Add new optional version argument
to the doctor examine command. This can be used to enable /
disable validation when examining older zip directories.
@fqazi fqazi force-pushed the validationVersion branch from 1996708 to 8d3c8a9 Compare February 7, 2022 20:17
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, and @postamar)


pkg/sql/catalog/desctestutils/descriptor_test_utils.go, line 35 at r1 (raw file):

Previously, ajwerner wrote…

Having reviewed a lot of this change, I wonder to what extent this plumbing here is making things a lot more painful. Should we make a variant which takes a version and just hard-code it to the latest version in the existing variant and plumb that through to the new variant? It feels like it would reduce the number of changes dramatically.

Done.

Made this the default behaviour which reduces the number of code changes by a lot.

@fqazi fqazi requested a review from ajwerner February 7, 2022 20:18
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 22 of 146 files at r1, 1 of 1 files at r2, 80 of 108 files at r3, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner, @dt, @fqazi, and @postamar)


pkg/ccl/changefeedccl/changefeed_stmt.go, line 897 at r3 (raw file):

	ctx context.Context, execCfg *sql.ExecutorConfig, txn *kv.Txn, desc catalog.TableDescriptor,
) (string, error) {
	col := execCfg.CollectionFactory.MakeCollection(ctx, nil)

nit: add back comment


pkg/ccl/changefeedccl/rowfetcher_cache.go, line 74 at r3 (raw file):

		codec:      codec,
		leaseMgr:   leaseMgr,
		collection: cf.NewCollection(nil /* TemporarySchemaProvider */),

nit: add back comment


pkg/cli/doctor.go, line 96 at r3 (raw file):

that may not exist on downlevel versions.
`,
		Args: cobra.MinimumNArgs(1),

nit: cobra.RangeArgs(1, 2)


pkg/sql/create_view.go, line 209 at r3 (raw file):

		// currently relied on in import and restore code and tests.
		var creationTime hlc.Timestamp
		desc, err := makeViewTableDesc(params.ctx, viewName, n.viewQuery, n.dbDesc.GetID(), schema.GetID(), id, n.columns, creationTime, privs, &params.p.semaCtx, params.p.EvalContext(), params.p.EvalContext().Settings, n.persistence, n.dbDesc.IsMultiRegion(), params.p)

nit: consider wrapping this


pkg/sql/temporary_schema.go, line 524 at r3 (raw file):

	// Build a set of all databases with temporary objects.
	var allDbDescs []catalog.DatabaseDescriptor
	descsCol := c.collectionFactory.NewCollection(ctx, nil)

nit: add back comment


pkg/sql/virtual_schema.go, line 239 at r3 (raw file):

) (descpb.TableDescriptor, error) {
	stmt, err := parser.ParseOne(v.schema)

nit: newline


pkg/sql/virtual_schema.go, line 254 at r3 (raw file):

		create.Name.Table(),
		tree.AsStringWithFlags(create.AsSource, tree.FmtParsable),
		0, /* parentID */

nit: add back comments

Previously, the descriptor validation code did not
take a version number, so it was not possible to version
gate new validation logic. This was inadequate because
when new fields are introduced we don't want their validation
to kick in for certain cases like the debug doctor, such as any
new fields with non-zero defaults. To address this, this patch add supports
for version numbers inside the validation, and updates unit tests
to pass this in as well. It also adds a new option on debug doctor
to run a version of validation.

Release note (cli change): Add new optional version argument
to the doctor examine command. This can be used to enable /
disable validation when examining older zip directories.
@fqazi fqazi force-pushed the validationVersion branch from 8d3c8a9 to b181614 Compare February 7, 2022 22:06
@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Feb 8, 2022

bors r+

@craig craig bot merged commit a7b1c17 into cockroachdb:master Feb 8, 2022
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Feb 8, 2022

Build succeeded:

RajivTS pushed a commit to RajivTS/cockroach that referenced this pull request Mar 6, 2022
Previously, the descriptor validation code did not
take a version number, so it was not possible to version
gate new validation logic. This was inadequate because
when new fields are introduced we don't want their validation
to kick in for certain cases like the debug doctor, such as any
new fields with non-zero defaults. To address this, this patch add supports
for version numbers inside the validation, and updates unit tests
to pass this in as well. It also adds a new option on debug doctor
to run a version of validation.

Release note (cli change): Add new optional version argument
to the doctor examine command. This can be used to enable /
disable validation when examining older zip directories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants