Skip to content

sql: use declarative schemachange for add column sequence exprs#81782

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:expressionDefaultValues
Jun 7, 2022
Merged

sql: use declarative schemachange for add column sequence exprs#81782
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:expressionDefaultValues

Conversation

@fqazi
Copy link
Copy Markdown
Collaborator

@fqazi fqazi commented May 24, 2022

Fixes: #81781

Previously, the declarative schema changer was disabled for
add column expressions with sequence references (i.e. default,
on update, computed) because we were missing telemetry
from the legacy schema changer for during backfill related failures.
This was inadequate because the lack of telemetry from these
areas could be seen as a usability issue. To address this,
this patch adds in support for missing telemetry and enables
supports for add column operations with sequence operations.

Release note: None

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@fqazi fqazi force-pushed the expressionDefaultValues branch from 5e63e21 to a0836b4 Compare May 25, 2022 03:39
@fqazi fqazi marked this pull request as ready for review May 25, 2022 03:39
@fqazi fqazi requested a review from a team May 25, 2022 03:39
Copy link
Copy Markdown
Contributor

@Xiang-Gu Xiang-Gu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @fqazi and @Xiang-Gu)


pkg/sql/schemachanger/scexec/exec_backfill.go line 256 at r1 (raw file):

	}); err != nil {
		// We ran into an  uncategorized schema change error.
		deps.Telemetry().IncrementSchemaChangeErrorType("uncategorized")

so all we need to do is add a Telemetry interface for declarative schema changer, and invoke it during backfill if the backfill fails? I saw only the "uncategorized" error. Are there any other types of errors?

Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's something funny going on in the restore case

Reviewed 6 of 7 files at r1, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @fqazi, and @Xiang-Gu)


-- commits line 17 at r1:
detritus

@fqazi fqazi force-pushed the expressionDefaultValues branch 4 times, most recently from 0493186 to f703132 Compare May 26, 2022 16:57
@fqazi fqazi requested review from a team and dt and removed request for a team May 26, 2022 16:57
@fqazi fqazi force-pushed the expressionDefaultValues branch from f703132 to 01c2118 Compare May 26, 2022 16:58
@fqazi fqazi requested a review from ajwerner May 26, 2022 17:00
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, and @Xiang-Gu)


-- commits line 17 at r1:

Previously, ajwerner wrote…

detritus

Done.


pkg/sql/schemachanger/scexec/exec_backfill.go line 256 at r1 (raw file):

Previously, Xiang-Gu (Xiang Gu) wrote…

so all we need to do is add a Telemetry interface for declarative schema changer, and invoke it during backfill if the backfill fails? I saw only the "uncategorized" error. Are there any other types of errors?

The only other type we have is for constraint validation failures. These can be extended later on

@fqazi fqazi requested a review from Xiang-Gu May 31, 2022 14:29
@fqazi fqazi force-pushed the expressionDefaultValues branch 2 times, most recently from 7e8fda1 to f9bd08e Compare June 2, 2022 22:11
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very close, nice

Reviewed 2 of 9 files at r3.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, @fqazi, and @Xiang-Gu)


pkg/sql/catalog/rewrite/rewrite.go line 397 at r3 (raw file):

}

// SchemaChangerStateInDescs goes over all mutable descriptors and cleans any

Can you expand this commentary to make it more clear what this does? I feel like the name could also be clearer. How about MaybeClearSchemaChangerStateInDescs? Explain why it exists and how it ought to be used.

Code quote:

// SchemaChangerStateInDescs goes over all mutable descriptors and cleans any
// empty state information.
func SchemaChangerStateInDescs(descriptors []catalog.MutableDescriptor) error {

pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_add_column.go line 170 at r3 (raw file):

	if desc.HasDefault() {
		expression := b.WrapExpression(tbl.TableID, cdd.DefaultExpr)
		// Sequence references inside expressions are unsupported, since these will

Is this no longer true?


pkg/sql/schemachanger/testdata/alter_table_add_column line 15 at r3 (raw file):

test
ALTER TABLE db.public.tbl ADD COLUMN j INT NOT NULL DEFAULT nextval('db.public.sq1')
----

nit: add a separate test

@fqazi fqazi force-pushed the expressionDefaultValues branch from f9bd08e to babf830 Compare June 6, 2022 16:33
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @dt, and @Xiang-Gu)


pkg/sql/catalog/rewrite/rewrite.go line 397 at r3 (raw file):

Previously, ajwerner wrote…

Can you expand this commentary to make it more clear what this does? I feel like the name could also be clearer. How about MaybeClearSchemaChangerStateInDescs? Explain why it exists and how it ought to be used.

// MaybeClearSchemaChangerStateInDescs goes over all mutable descriptors and
// cleans any state information from descriptors which have no targets associated
// with the corresponding jobs. Not having any targets in an individual state
// is not enough, since the descriptor in question could be a dependent (for
// example a type reference in a wider schema change).


pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_add_column.go line 170 at r3 (raw file):

Previously, ajwerner wrote…

Is this no longer true?

Yes, this limitation is now lifted, we allowed sequence expressions inside default expressions.


pkg/sql/schemachanger/testdata/alter_table_add_column line 15 at r3 (raw file):

Previously, ajwerner wrote…

nit: add a separate test

Done

@fqazi fqazi requested a review from ajwerner June 6, 2022 20:54
@fqazi fqazi force-pushed the expressionDefaultValues branch from babf830 to 760f93a Compare June 7, 2022 03:08
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 4 files at r2, 6 of 9 files at r3, 3 of 3 files at r4, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @dt, @fqazi, and @Xiang-Gu)


pkg/sql/schemachanger/testdata/alter_table_add_column line 2049 at r4 (raw file):

         - i
  -      name: crdb_internal_index_7_name_placeholder
  +      name: tbl_pkey

Should this still be called tbl_pkey here?

Fixes: cockroachdb#81781

Previously, the declarative schema changer was disabled for
add column expressions with sequence references (i.e. default,
on update, computed) because we were missing telemetry
from the legacy schema changer for during backfill related failures.
This was inadequate because the lack of telemetry from these
areas could be seen as a usability issue. To address this,
this patch adds in support for missing telemetry and enables
supports for add column operations with sequence operations.

Release note: None
@fqazi fqazi force-pushed the expressionDefaultValues branch from 760f93a to 9a9fe00 Compare June 7, 2022 03:54
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner, @dt, and @Xiang-Gu)


pkg/sql/schemachanger/testdata/alter_table_add_column line 2049 at r4 (raw file):

Previously, ajwerner wrote…

Should this still be called tbl_pkey here?

That does look fishy, the existing rules have this issue. Let me handle this in the other PR where I'm adding UNIQUE support.

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Jun 7, 2022

@ajwerner TFTR!

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Jun 7, 2022

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 7, 2022

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 7, 2022

Build failed:

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Jun 7, 2022

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 7, 2022

Build failed:

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Jun 7, 2022

bors r+

@craig craig bot merged commit 02cc393 into cockroachdb:master Jun 7, 2022
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 7, 2022

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sql/schemachanger: use declarative schemachange for add column sequence expr

4 participants