Skip to content

changefeedccl: improve commit-to-emit row latency #36289

@danhhz

Description

@danhhz

The promise of RangeFeed is that we will push row changes from raft all the way out to the sink, which should result in very low end-to-end latencies for this. (Note: resolved timestamp latency, which is the only thing we test as of 2019-03-28, will always be bounded by closed timestamps and so is expected to be larger). However, this promise of low latency doesn't work in practice because of 2 small-ish issues.

  • We currently wait for the TableHistory to be above the row timestamp, so that we're always guaranteed to emit each row with the very latest possible TableDescriptor. Specifically, during the second half of a schema change (e.g. adding a row), we always want to interpret a changed kv with the descriptor where the mutation is live so that the new column is in the output. An alternate is to switch TableHistory to use RangeFeed and stop blocking on the latest descriptor, instead using any descriptor that's valid to lease for the kv's updated timestamp. Then, if we see a descriptor where a mutation goes live, restart the poller from that descriptor's first valid timestamp (the modification timestamp) and re-emit any changes. This trades a small window of duplicates during a schema change (rare) for dramatically lower latency in the common case.
  • We also need to make this retry with backoff into something based on signals. https://github.com/cockroachdb/cockroach/blob/master/pkg/ccl/changefeedccl/buffer.go#L218-L238

Jira issue: CRDB-4509

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-cdcChange Data CaptureC-performancePerf of queries or internals. Solution not expected to change functional behavior.T-cdcX-staleno-issue-activity

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions