Skip to content

streamingccl: add generation support for sinkless stream#67189

Closed
annezhu98 wants to merge 1 commit intocockroachdb:masterfrom
annezhu98:57406
Closed

streamingccl: add generation support for sinkless stream#67189
annezhu98 wants to merge 1 commit intocockroachdb:masterfrom
annezhu98:57406

Conversation

@annezhu98
Copy link
Copy Markdown

@annezhu98 annezhu98 commented Jul 2, 2021

See rfc for more context.

Part of Epic CRDB-6407

Informs: #57406

Release note: None

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@annezhu98 annezhu98 requested review from adityamaru and pbardea July 2, 2021 18:44
@annezhu98 annezhu98 changed the title streamingccl: add GenerationEvent type [WIP] streamingccl: add GenerationEvent type Jul 2, 2021
@annezhu98 annezhu98 changed the title [WIP] streamingccl: add GenerationEvent type [WIP] streamingccl: add support for generations for sinkless streams Jul 2, 2021
@annezhu98 annezhu98 force-pushed the 57406 branch 2 times, most recently from c6cf1e7 to c3b9aa3 Compare July 2, 2021 19:36
@annezhu98 annezhu98 changed the title [WIP] streamingccl: add support for generations for sinkless streams [WIP] streamingccl: add generation support for sinkless stream Jul 2, 2021
@annezhu98 annezhu98 force-pushed the 57406 branch 2 times, most recently from defdc48 to 70fd867 Compare July 14, 2021 20:09
@annezhu98 annezhu98 force-pushed the 57406 branch 7 times, most recently from a13a0d4 to aa02eb1 Compare July 21, 2021 20:22
@annezhu98 annezhu98 marked this pull request as ready for review July 21, 2021 20:26
@annezhu98 annezhu98 requested review from a team July 21, 2021 20:26
@annezhu98 annezhu98 changed the title [WIP] streamingccl: add generation support for sinkless stream streamingccl: add generation support for sinkless stream Jul 21, 2021
craig bot pushed a commit that referenced this pull request Aug 10, 2021
…68660 #68661

67090: sql: periodically flush sqlstats r=Azhng a=Azhng

Previous PR: #67866

This commit introduces a new persisted sql stats subsystem
that wraps the existing node-local sql stats subsystem.
This new subsystem is responsible for flushing the in-meory
statistics into the system table periodically, or when it
detects memory pressure.
This replaces sql.Server's in-memory sqlStats provider.

Release note (sql change): SQL stats now can be persisted into
system.statement_statistics and system.transaction_statistics
tables by enabling the sql.stats.flush.enable cluster setting.
The interval of persistence is determined by the new
sql.stats.flush.interval cluster setting which defaults to 1 hour.

68195: streamingccl: allow stream ingestion processors to keep running on `GenerationEvent` r=annezhu98 a=annezhu98

Previously, a stream ingestion processor would shut down if it ever loses connection with its stream client. With generation support, the processor should not immediately move to draining state, instead, it should be in `StateRunning` to poll for cutover signal sent by the coordinator. Generation support will be implemented by the following PR: #67189

The first commit adds `GenerationEvent` as a valid event type that can be emitted over a cluster stream.
The second commit implements the mechanism that keeps processors running when losing connection with the client.

68288: changefeedccl: Propagate pushback throughout changefeed pipeline. r=ajwerner a=miretskiy

Propagate pushback information throughout changefeed pipeline.

The pushback propagation is accomplished by associating a `Resource`
with each event that is processed by changefeed system.  The resource is
propagated throughout changefeed, and is released when the event
has been written to the sink.

This change also updates and simplifies event memory accounting.
Prior to this PR, memory accounting was incomplete and was error prone.
This PR simplifies memory accounting by allocating resources once when
the event arrives (blocking buffer), and releasing resources
when event is written to the sink.  Dynamic modifications to the amount
of allocated memory are, in general, not safe (without additional
complexity).  To accommodate the fact that during event processing we
use more memory (e.g. to parse and convert this event), we over-allocate
resources to the the event.

Release Notes: Enterprise change; changefeed will slow down correctly
whenever there is a slow down in the system (i.e. downstream sink is
slow).

68444: multiregionccl: add zone config waiting for partitioned tables r=arulajmani a=pawalt

Previously, wait-for-zone-configs could only wait on tables, meaning we
couldn't wait for zone config changes to apply to REGIONAL BY ROW
partitions. This PR adds a `partition-name` flag to allow users to wait
on a specific partition.

Release note: None

A future PR will extend the tracing to allow us to trace REGIONAL BY ROW queries, but those changes combined with these will likely be too large for a single PR.

68596: authors: add pseudomuto to authors r=pseudomuto a=pseudomuto

Release note: None

68611: ci: move bazel build scripts to `build/teamcity` directory r=rail a=rickystewart

This gives a little more structure to what was previously a completely
flat `build` directory that contained one or more scripts for every
single build configuration.

When this PR is merged I'll move the affected build configurations to
final destinations in the `Cockroach / CI` subproject in TC.

Release note: None

68614: server: reduce flakiness of TestEnsureSQLStatsAreFlushedForTelemetry r=Azhng a=Azhng

Previously, this unit tests generates statements with the fingerprint
'INSERT INTO _ VALUES (_)'. However, this is a very common fingerprint
and can collide with actual statements issued by other subsystems, which
in turn cause test failure.
This commit change the test cases to use uncommon statements that should
not cause fingerprint collision

Resolves #66826

Release note: None

68637: kvserver/closedts: fix nil dereference in HTML generation r=aliher1911 a=erikgrinaker

The `/debug/closedts-{sender,receiver}` endpoints could panic due to a
nil dereference if the last update in the sidetransport buffer was
removed by the time it was rendered. This patch adds a `nil` check to
avoid that panic.

Release note (bug fix): Fixed a crash in the
`/debug/closedts-{sender,receiver}` advanced debug pages if the last
message of the closed timestamp side transport buffer was removed before
rendering.

68642: kvserver: deflake TestNonVoterCatchesUpViaRaftSnapshotQueue r=aayushshah15 a=aayushshah15

Fixes #68142

Release note: None

68660: authors: Add shiranka to authors r=samiskin a=samiskin



68661: add ding@ to AUTHORS r=davidwding a=davidwding



Co-authored-by: Azhng <archer.xn@gmail.com>
Co-authored-by: Anne Zhu <anne.zhu@cockroachlabs.com>
Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com>
Co-authored-by: Peyton Walters <peyton.walters@cockroachlabs.com>
Co-authored-by: David Muto (pseudomuto) <david.muto@gmail.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
Co-authored-by: Aayush Shah <aayush.shah15@gmail.com>
Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com>
Co-authored-by: David Ding <ding@cockroachlabs.com>
@annezhu98 annezhu98 force-pushed the 57406 branch 2 times, most recently from 732892d to 2903997 Compare August 11, 2021 16:43
Once the coordinator node receives a `GenerationEvent` from its client, dist flow will be replanned with the topology associated with the new generation.

Release note: None
@cockroach-teamcity
Copy link
Copy Markdown
Member

cockroach-teamcity commented Aug 24, 2021

CLA assistant check
All committers have signed the CLA.

@pbardea pbardea removed their request for review August 31, 2021 14:07
@rafiss rafiss removed the request for review from a team April 26, 2023 20:04
@adityamaru adityamaru closed this Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants