Skip to content

streaming: ingestion jobs should be able to transition between stream "generations" #57406

@pbardea

Description

@pbardea

The streaming ingestion job should be able to transition to reading from a new generation when a new one advertises itself.

A stream may be composed of several generations. A generation is defined by a start-time which indicates the time at which changes started being emitted to that particular generation. A stream might periodically produce a new generation (when a new generation is created, it will eventually stop writing to the old one).

Upon notification of a new generation, the ingestion job should plan a new flow for the new generation and pickup ingesting the stream.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-disaster-recoveryC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions