-
Notifications
You must be signed in to change notification settings - Fork 4.1k
changefeeds: Implement trailing PTS handling #76247
Copy link
Copy link
Closed
Labels
A-cdcChange Data CaptureChange Data CaptureC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc
Description
Currently, changefeed relies on laying PTS record during backfills only. After backfill
completion, the record is released.
This behavior has few unfortunate properties:
- A recurring, transient error (e.g unavailable downstream sink) with
changefeed running with "protect gc on pause" and on_error pause options not be able to protect data if the
error keeps happening for too long. - We cannot safely run changefeeds on tables with low gc ttl.
- We cannot safely run changefeeds on (soon to be implemented) ephemeral tables.
- We cannot safely run in serverless environment.
As such, we should change the current PTS management to be much simpler and address
above deficiencies: Namely, PTS record should be created when changefeed starts, and thereafter advanced
with certain periodicity (e.g. checkpoint).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-cdcChange Data CaptureChange Data CaptureC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc