streamingccl: add generation support for sinkless stream#67189
Closed
annezhu98 wants to merge 1 commit intocockroachdb:masterfrom
Closed
streamingccl: add generation support for sinkless stream#67189annezhu98 wants to merge 1 commit intocockroachdb:masterfrom
annezhu98 wants to merge 1 commit intocockroachdb:masterfrom
Conversation
Member
c6cf1e7 to
c3b9aa3
Compare
defdc48 to
70fd867
Compare
a13a0d4 to
aa02eb1
Compare
craig bot
pushed a commit
that referenced
this pull request
Aug 10, 2021
…68660 #68661 67090: sql: periodically flush sqlstats r=Azhng a=Azhng Previous PR: #67866 This commit introduces a new persisted sql stats subsystem that wraps the existing node-local sql stats subsystem. This new subsystem is responsible for flushing the in-meory statistics into the system table periodically, or when it detects memory pressure. This replaces sql.Server's in-memory sqlStats provider. Release note (sql change): SQL stats now can be persisted into system.statement_statistics and system.transaction_statistics tables by enabling the sql.stats.flush.enable cluster setting. The interval of persistence is determined by the new sql.stats.flush.interval cluster setting which defaults to 1 hour. 68195: streamingccl: allow stream ingestion processors to keep running on `GenerationEvent` r=annezhu98 a=annezhu98 Previously, a stream ingestion processor would shut down if it ever loses connection with its stream client. With generation support, the processor should not immediately move to draining state, instead, it should be in `StateRunning` to poll for cutover signal sent by the coordinator. Generation support will be implemented by the following PR: #67189 The first commit adds `GenerationEvent` as a valid event type that can be emitted over a cluster stream. The second commit implements the mechanism that keeps processors running when losing connection with the client. 68288: changefeedccl: Propagate pushback throughout changefeed pipeline. r=ajwerner a=miretskiy Propagate pushback information throughout changefeed pipeline. The pushback propagation is accomplished by associating a `Resource` with each event that is processed by changefeed system. The resource is propagated throughout changefeed, and is released when the event has been written to the sink. This change also updates and simplifies event memory accounting. Prior to this PR, memory accounting was incomplete and was error prone. This PR simplifies memory accounting by allocating resources once when the event arrives (blocking buffer), and releasing resources when event is written to the sink. Dynamic modifications to the amount of allocated memory are, in general, not safe (without additional complexity). To accommodate the fact that during event processing we use more memory (e.g. to parse and convert this event), we over-allocate resources to the the event. Release Notes: Enterprise change; changefeed will slow down correctly whenever there is a slow down in the system (i.e. downstream sink is slow). 68444: multiregionccl: add zone config waiting for partitioned tables r=arulajmani a=pawalt Previously, wait-for-zone-configs could only wait on tables, meaning we couldn't wait for zone config changes to apply to REGIONAL BY ROW partitions. This PR adds a `partition-name` flag to allow users to wait on a specific partition. Release note: None A future PR will extend the tracing to allow us to trace REGIONAL BY ROW queries, but those changes combined with these will likely be too large for a single PR. 68596: authors: add pseudomuto to authors r=pseudomuto a=pseudomuto Release note: None 68611: ci: move bazel build scripts to `build/teamcity` directory r=rail a=rickystewart This gives a little more structure to what was previously a completely flat `build` directory that contained one or more scripts for every single build configuration. When this PR is merged I'll move the affected build configurations to final destinations in the `Cockroach / CI` subproject in TC. Release note: None 68614: server: reduce flakiness of TestEnsureSQLStatsAreFlushedForTelemetry r=Azhng a=Azhng Previously, this unit tests generates statements with the fingerprint 'INSERT INTO _ VALUES (_)'. However, this is a very common fingerprint and can collide with actual statements issued by other subsystems, which in turn cause test failure. This commit change the test cases to use uncommon statements that should not cause fingerprint collision Resolves #66826 Release note: None 68637: kvserver/closedts: fix nil dereference in HTML generation r=aliher1911 a=erikgrinaker The `/debug/closedts-{sender,receiver}` endpoints could panic due to a nil dereference if the last update in the sidetransport buffer was removed by the time it was rendered. This patch adds a `nil` check to avoid that panic. Release note (bug fix): Fixed a crash in the `/debug/closedts-{sender,receiver}` advanced debug pages if the last message of the closed timestamp side transport buffer was removed before rendering. 68642: kvserver: deflake TestNonVoterCatchesUpViaRaftSnapshotQueue r=aayushshah15 a=aayushshah15 Fixes #68142 Release note: None 68660: authors: Add shiranka to authors r=samiskin a=samiskin 68661: add ding@ to AUTHORS r=davidwding a=davidwding Co-authored-by: Azhng <archer.xn@gmail.com> Co-authored-by: Anne Zhu <anne.zhu@cockroachlabs.com> Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com> Co-authored-by: Peyton Walters <peyton.walters@cockroachlabs.com> Co-authored-by: David Muto (pseudomuto) <david.muto@gmail.com> Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com> Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com> Co-authored-by: Aayush Shah <aayush.shah15@gmail.com> Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com> Co-authored-by: David Ding <ding@cockroachlabs.com>
732892d to
2903997
Compare
Once the coordinator node receives a `GenerationEvent` from its client, dist flow will be replanned with the topology associated with the new generation. Release note: None
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See rfc for more context.
Part of Epic CRDB-6407
Informs: #57406
Release note: None