Skip to content

changefeeds: Implement trailing PTS handling #76247

@miretskiy

Description

@miretskiy

Currently, changefeed relies on laying PTS record during backfills only. After backfill
completion, the record is released.

This behavior has few unfortunate properties:

  • A recurring, transient error (e.g unavailable downstream sink) with
    changefeed running with "protect gc on pause" and on_error pause options not be able to protect data if the
    error keeps happening for too long.
  • We cannot safely run changefeeds on tables with low gc ttl.
  • We cannot safely run changefeeds on (soon to be implemented) ephemeral tables.
  • We cannot safely run in serverless environment.

As such, we should change the current PTS management to be much simpler and address
above deficiencies: Namely, PTS record should be created when changefeed starts, and thereafter advanced
with certain periodicity (e.g. checkpoint).

Metadata

Metadata

Assignees

Labels

A-cdcChange Data CaptureC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions