-
Notifications
You must be signed in to change notification settings - Fork 4.1k
streamingccl: shift Cutover and GenerationEvent handling from processors to coordinator #68475
Description
Currently, every stream ingestion processor checks the ingestion job for a cutover signal and reacts appropriately. This means that every node in a 100 node cluster would periodically poll the job. Similarly, every ingestion processor also listens for a GenerationEvent from the client and has custom logic to handle such events.
As explained in #68195 (comment) we want to eventually move to a model where we eliminate these many to 1 relationships, and only have the coordinator aware of cutover and generation events. In other words, the custom logic to handle such special events should be abstracted away from the processors running on every node in the cluster. The processors should only be responsible for ingesting until told not to. When the coordinator sees a special event, it should push a signal through the stream ingestion DistSQL flow, and the processors should react to the signal appropriately.
The big blocker here is the ability to push a row from the coordinator into a running DistSQL flow. This will likely require some coordination with the SQL Queries team.
Epic CRDB-19048