-
Notifications
You must be signed in to change notification settings - Fork 4.1k
cdc: handle column renames consistently in cdc expressions #84767
Copy link
Copy link
Closed
Labels
A-cdcChange Data CaptureChange Data CaptureA-cdc-expressionsFeatures related to changefeed projections and filtersFeatures related to changefeed projections and filtersC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc
Description
Current behavior:
Column references are evaluated as of the event. That means that if you do
CREATE CHANGEFEED AS SELECT i FROM foo
ALTER TABLE foo RENAME COLUMN i to i_old, ADD COLUMN i intSubsequent events will use your new i column, not the old one.
If you do
CREATE CHANGEFEED AS SELECT i FROM foo
ALTER TABLE foo RENAME COLUMN i to i_old, ADD COLUMN i int DEFAULT 0The changefeed will immediately-ish fail, because when it tries to evaluate the projection as of the backfill operation it doesn't see either i column.
So action is definitely required. We need to do one of three things.
- "Compile" column references in expressions to column ids. This makes the behavior clearly defined and simple in terms of our implementation, but will also be potentially deeply confusing in its consequences.
- Or, make any rename of a column that is referenced by a changefeed expression (or ideally even used as a key to cdc_prev() if that's still a JSON function by the time you're reading this) cause the changefeed to fail (setting a schema change boundary like you dropped the table).
- Or, keep the current behavior but find and fix all bugs (adding a column with a default value shouldn't cause an error, cdc_prev() behavior needs to be very carefully tested and documented, ensure renames that change a referenced column's type can't cause a type assertion panic, etc.)
Jira issue: CRDB-17852
Epic CRDB-17161
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-cdcChange Data CaptureChange Data CaptureA-cdc-expressionsFeatures related to changefeed projections and filtersFeatures related to changefeed projections and filtersC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc