Skip to content

cdc: handle column renames consistently in cdc expressions #84767

@HonoreDB

Description

@HonoreDB

Current behavior:

Column references are evaluated as of the event. That means that if you do

CREATE CHANGEFEED AS SELECT i FROM foo
ALTER TABLE foo RENAME COLUMN i to i_old, ADD COLUMN i int

Subsequent events will use your new i column, not the old one.

If you do

CREATE CHANGEFEED AS SELECT i FROM foo
ALTER TABLE foo RENAME COLUMN i to i_old, ADD COLUMN i int DEFAULT 0

The changefeed will immediately-ish fail, because when it tries to evaluate the projection as of the backfill operation it doesn't see either i column.

So action is definitely required. We need to do one of three things.

  1. "Compile" column references in expressions to column ids. This makes the behavior clearly defined and simple in terms of our implementation, but will also be potentially deeply confusing in its consequences.
  2. Or, make any rename of a column that is referenced by a changefeed expression (or ideally even used as a key to cdc_prev() if that's still a JSON function by the time you're reading this) cause the changefeed to fail (setting a schema change boundary like you dropped the table).
  3. Or, keep the current behavior but find and fix all bugs (adding a column with a default value shouldn't cause an error, cdc_prev() behavior needs to be very carefully tested and documented, ensure renames that change a referenced column's type can't cause a type assertion panic, etc.)

Jira issue: CRDB-17852

Epic CRDB-17161

Metadata

Metadata

Assignees

Labels

A-cdcChange Data CaptureA-cdc-expressionsFeatures related to changefeed projections and filtersC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions