schemachanger: Enable adding/dropping path of check constraints#89778
Conversation
d33c0d4 to
4a412c6
Compare
ajwerner
left a comment
There was a problem hiding this comment.
This is close. I think the UNVALIDATED state is going to be interesting to handle. I wonder if @postamar has any ideas.
Reviewed 2 of 2 files at r1, 5 of 5 files at r2, 11 of 17 files at r3.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @Xiang-Gu)
pkg/sql/schemachanger/scexec/exec_validation.go line 85 at r3 (raw file):
return scerrors.SchemaChangerUserError(err) } constraint.CheckConstraint.Validity = descpb.ConstraintValidity_Validated
this doesn't actually do anything, right?
pkg/sql/schemachanger/scexec/scmutationexec/references.go line 177 at r3 (raw file):
// enqueuing a mutation. We need modify the next operation accordingly // bc such a change means it's possible we might no longer find // a mutation later.
Rather, I think it should be an error if the constraint is unvalidated and we should use a different op to make an unvalidated constraint absent. There's some interesting questions on how to model this. It's like we have a more complex state machine here than we know how to deal with.
PUBLIC->VALIDATED->VALIDATING->ABSENT We can go from PUBLIC to VALIDATING in one transaction I believe.
ABSENT->VALIDATING->VALIDATED->PUBLIC
UNVALIDATED->VALIDATING->VALIDATED->PUBLIC
PUBLIC->UNVALIDATED
ABSENT->UNVALIDATED
UNVALIDATED->ABSENT
Code quote:
// TODO (xiang): We ought to check whether the check constraint
// is Unvalidated. If so, we can drop it immediately without
// enqueuing a mutation. We need modify the next operation accordingly
// bc such a change means it's possible we might no longer find
// a mutation later.4a412c6 to
2f7bdaf
Compare
postamar
left a comment
There was a problem hiding this comment.
My solution to the UNVALIDATED problem would be to have two check constraint elements: CheckConstraint and CheckConstraintUnvalidated:
- ADD CONSTRAINT results in a
CheckConstraintpublic target; - ADD CONSTRAINT ... NOT VALID results in a
CheckConstraintUnvalidatedpublic target; - VALIDATE CONSTRAINT results in a
CheckConstraintUnvalidatedabsent target and aCheckConstraintpublic target.
2f7bdaf to
be6a2bc
Compare
|
After spending some time on several releated issues and merging their fix, I think this PR is RFAL again. |
|
So, this all looks good, but this code doesn't actually get exercised much so how do we know if it actually works for real? DROP TABLE will drop check constraints but the whole table is dropped so that's that. Sure, dropping a column that's referenced in a check constraint will also drop the check constraint, but this doesn't strike me as something that has a lot of test coverage, whether in I think this PR needs to add support for ALTER TABLE ... ADD CONSTRAINT ... CHECK and the corresponding DROP CONSTRAINT in the builder, or something equivalent that guarantees these paths are exercised a fair bit by default in all tests in the codebase. We've seen this in the past with the declarative schema changer: until it's on by default we can't be confident that it works. |
Xiang-Gu
left a comment
There was a problem hiding this comment.
@postamar I acknowledged that. I am planning to start supporting ALTER TABLE ... ADD CONSTRAINT ... CHECK in a follow-up PR, because this PR itself is getting big enough and I don't want it to be even bigger. In that follow-up PR we will be able to exercise those adding/dropping path more heavily.
There will be one (or two) PR after this to similarly enable adding/dropping path for ConstraintName and ConstraintComment. Then we will be good to start implementing ADD CONSTRAINT.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner and @postamar)
pkg/sql/schemachanger/scexec/exec_validation.go line 85 at r3 (raw file):
Previously, ajwerner wrote…
this doesn't actually do anything, right?
Correct, I don't think it does anything. In fact, I think in the old schema changer, we didn't even set this field after validating a check constraint:
Lines 714 to 816 in 397d5da
I added it simply bc I think this is a sane thing to do.
Code quote:
func executeValidateCheckConstraint(
ctx context.Context, deps Dependencies, op *scop.ValidateCheckConstraint,
) error {be6a2bc to
a0d18c3
Compare
Xiang-Gu
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner and @postamar)
pkg/sql/schemachanger/scexec/exec_validation.go line 85 at r3 (raw file):
Previously, Xiang-Gu (Xiang Gu) wrote…
Correct, I don't think it does anything. In fact, I think in the old schema changer, we didn't even set this field after validating a check constraint:
Lines 714 to 816 in 397d5da
I added it simply bc I think this is a sane thing to do.
Please disregard what I said above. you are right -- this is a validation stage and hence any changes to the descriptor won't be persisted. I've removed that line.
Ok, that's fine. |
|
CI is green and I think it's now RFAL! |
postamar
left a comment
There was a problem hiding this comment.
This is close, only minor comments.
| } | ||
| }), | ||
| // TODO (xiang): Add an operation that updates back-references in potentially | ||
| // affected types and sequences when implementing `add constraint`. |
There was a problem hiding this comment.
I believe it simply involves emitting UpdateTableBackReferencesInTypes and UpdateBackReferencesInSequences operations just like on the dropping path, in which case you might as well do so right now.
| // name, etc. disappear once the constraint reaches a suitable state. | ||
| // TODO (xiang): The dep rules here are not complete, as they are aimed specifically | ||
| // for check constraints only. Complete them when we properly support the | ||
| // other two constraints: UniqueWithIndex and ForeignKey. |
There was a problem hiding this comment.
Actually there's foreign keys, index-backed uniques, and also UniqueWithoutIndex as well as not-NULL.
| return false | ||
| } | ||
|
|
||
| func isCheckConstraint(e scpb.Element) bool { |
There was a problem hiding this comment.
Wouldn't this be a good place for a TODO to expand this predicate to include all non-index-backed constraints?
| return func(e scpb.Element) bool { | ||
| return !predicate(e) | ||
| } | ||
| } |
There was a problem hiding this comment.
This might not be useful right now, but it keeps coming up from time to time. Let's leave it here for the time being?
There was a problem hiding this comment.
It will cause a CI linter failure -- object rules.not is not used -- as happened in this run.
Is there easy way, like a directive, to skip?
| constraint.targetStatus(scpb.ToAbsent), | ||
| constraint.currentStatus( | ||
| scpb.Status_PUBLIC, | ||
| ), |
| } | ||
| }), | ||
| ), | ||
| equiv(scpb.Status_WRITE_ONLY), |
There was a problem hiding this comment.
Shouldn't revertible(false) apply to WRITE_ONLY?
| ConstraintID: this.ConstraintID, | ||
| } | ||
| }), | ||
| ), |
There was a problem hiding this comment.
Since this is revertible, please add a test case which covers rolling back a drop after PUBLIC -> VALIDATED (the rollback will exercise VALIDATED -> PUBLIC on the adding path), if nothing else.
a0d18c3 to
08e0ab7
Compare
Xiang-Gu
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner and @postamar)
pkg/sql/schemachanger/scplan/internal/opgen/opgen_check_constraint.go line 33 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
I believe it simply involves emitting
UpdateTableBackReferencesInTypesandUpdateBackReferencesInSequencesoperations just like on the dropping path, in which case you might as well do so right now.
done
pkg/sql/schemachanger/scplan/internal/opgen/opgen_check_constraint.go line 61 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
Since this is revertible, please add a test case which covers rolling back a drop after PUBLIC -> VALIDATED (the rollback will exercise VALIDATED -> PUBLIC on the adding path), if nothing else.
I think this will be added in the follow-up PR where we implement add constraint ... check
pkg/sql/schemachanger/scplan/internal/opgen/opgen_check_constraint.go line 62 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
Shouldn't
revertible(false)apply toWRITE_ONLY?
My understanding is that since we used equiv(write_only), we will get a transition write_only --> validated with no operations when we build this opgen. Later, in dep_two_version.go I think any transition with no operations will be skipped/collapsed. So in practice, a CheckConstraint element will never be in write_only status in a dropping path (maybe this is the beauty of equiv? We can, for all intents and purposes, just ignore that status and pretend it does not exist).
pkg/sql/schemachanger/scplan/internal/rules/dep_drop_constraint.go line 23 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
Actually there's foreign keys, index-backed uniques, and also UniqueWithoutIndex as well as not-NULL.
Ahh, yeah. Changed the comments.
pkg/sql/schemachanger/scplan/internal/rules/helpers.go line 372 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
Wouldn't this be a good place for a TODO to expand this predicate to include all non-index-backed constraints?
done. We had the same thought that we should program in a way that unifies our tretement for all the constraints that do not involve an index.
I renamed it to isSupportedNonIndexBackedConstraint.
pkg/sql/schemachanger/scplan/internal/rules/op_drop.go line 183 at r10 (raw file):
Previously, postamar (Marius Posta) wrote…
nit: whitespace
I wasn't able to see any formatting nits related to whitespace. Minding explaining a bit more?
08e0ab7 to
162bc59
Compare
Xiang-Gu
left a comment
There was a problem hiding this comment.
I've addressed your comments and left a few questions. Would appreciate it if you can take another look at it.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner and @postamar)
postamar
left a comment
There was a problem hiding this comment.
LGTM, though, again, it's hard to tell whether any of this actually works.
I think this will be added in the follow-up PR where we implement add constraint ... check
OK, I guess.
We can, for all intents and purposes, just ignore that status and pretend it does not exist).
If the tests pass (not saying they won't) then I have no problem with that.
I wasn't able to see any formatting nits related to whitespace. Minding explaining a bit more?
These files in rules have a uniformity of style that makes it easier to read what's otherwise quite information-dense code. Ideally we'd enforce this style with a linter rule, of course, but we don't.
1. Fixed a careless bug when removing a check constraint; 2. a minor rewrite to `isColumn` to make it more compact.
We added a field bool field, `FromHashShardedColumn`, in the CheckConstraint element protobuf. This will be needed in the future when we translate it to a descpb.CheckConstraint when doing check constraint related work.
We enable the adding/dropping path of check constraints in declarative schema changer (previously it changes between PUBLIC and ABSENT status with a NotImplemented operation). Namely, we introduced a new element status `validating` and reused `validated` such that the status transitions of a check constraint element is ABSENT <==> WRITE_ONLY <==> VALIDATED <==> PUBLIC and it's subject to the 2-version invariant. Correspondingly, we emit operations for each transition as follows: Adding path: - ABSENT ==> WRITE_ONLY: Add a check constraint (ADD direction) to the mutation slice (Mutation Type op) - WRITE_ONLY ==> VALIDATED: validate a check constraint (Validation Type op) - VALIDATED ==> PUBLIC: complete the check constraint adding mutation (Mutation Type op) Dropping path: - PUBLIC ==> VALIDATED: Add a check constraint (DROP direction) to the mutation slice (Mutation Type op) - VALIDATED ==> WRITE_ONLY: equivalent status - (*) WRITE_ONLY ==> ABSENT: complete the check constraint dropping mutation (Mutation Type op) A check constraint in - WRITE_ONLY means it will be enforced for all new UPDATE/INSERT. - VALIDATED means validation has succeeded but it's not yet public. N.B.: * means it's information publishing/destroying stage and hence should be non-revertible.
162bc59 to
a7f3ae6
Compare
|
TFTR! bors r+ |
|
Build succeeded: |
See each commit message for deatils.
Commit 1 (easy to review): Fixed a careless bug in existing code;
Commit 2 (easy to review): Added a bool field,
FromHashShardedIndex, in the protobufdefinition of the
CheckConstraintelement.Commit 3 (meaty commit): Enable adding/dropping path of check constraints.
Informs #89665