changefeedccl: mark avro format as no longer experimental#68818
changefeedccl: mark avro format as no longer experimental#68818craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
62b2447 to
514ad6a
Compare
0875f46 to
771af3a
Compare
771af3a to
2872f8d
Compare
miretskiy
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @HonoreDB, @spiffyyeng, and @stevendanna)
pkg/ccl/changefeedccl/changefeed_stmt.go, line 159 at r1 (raw file):
if opts[changefeedbase.OptFormat] == changefeedbase.DeprecatedOptFormatAvro { //TODO(zinger): See if there's a way to make this work with sinkless
i guess no... i'd just drop this todo...
pkg/ccl/changefeedccl/helpers_test.go, line 654 at r1 (raw file):
// a lot of cdc test framework, use with caution. Driver-agnostic tools don't // have clean ways of inspecting incoming notices. func expectNotice(t *testing.T, s serverutils.TestServerInterface, sql string, expected string) {
i won't mind if this very minor detail was left outside of tests...
Like: try it manually; see if you get the notice and don't blow anything up... I'm not sure i'd necessarily test for that.
a2d0f77 to
506ef41
Compare
|
bors r=[miretskiy, spiffyeng] |
HonoreDB
left a comment
There was a problem hiding this comment.
Thanks for the reviews!
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @miretskiy and @stevendanna)
pkg/ccl/changefeedccl/changefeed_stmt.go, line 159 at r1 (raw file):
Previously, miretskiy (Yevgeniy Miretskiy) wrote…
i guess no... i'd just drop this todo...
Ah well. Done.
pkg/ccl/changefeedccl/helpers_test.go, line 654 at r1 (raw file):
Previously, miretskiy (Yevgeniy Miretskiy) wrote…
i won't mind if this very minor detail was left outside of tests...
Like: try it manually; see if you get the notice and don't blow anything up... I'm not sure i'd necessarily test for that.
Yeah, testing this was one of those things where I thought it'd be easy and it ended up taking way too long and being awkward but at that point I was emotionally committed. I usually just grumble and delete the code when that happens but eh, it's a cheap test.
|
Build failed: |
The avro format for changefeeds now supports all column types and has been in production use for several releases. We'll now allow format=avro rather than format=experimental_avro The old string will remain supported because job payloads can persist across upgrades and downgrades. Release note (enterprise change): changefeed avro format no longer marked experimental
506ef41 to
69e70e7
Compare
|
bors retry |
|
Build failed (retrying...): |
|
This PR was included in a batch that was canceled, it will be automatically retried |
66889: jobs: retry jobs with exponential backoff r=ajwerner a=sajjadrizvi This commit adds a mechanism to retry jobs with exponentially increasing delays. This is achieved through two new columns in system.jobs table, last_run and num_runs. In addition, this commit adds cluster settings to control exponential-backoff parameters, initial delay and max delay, with corresponding settings `jobs.registry.retry.initial_delay` and `jobs.registry.retry.max_delay`. Finally, this commit adds a new partial-index in the jobs table that improves the performance of periodic queries run by registry in each node. Release note (general change): The behavior for retrying jobs, which fail due to a retriable error or due to job coordinator failure, is now delayed using exponential backoff. Before this change, jobs which failed in a retryable manner, would be resumed immediately on a different coordinator. This change reduces the impact of recurrently failing jobs on the cluster. This change adds two new cluster settings that control this behavior: "jobs.registry.retry.initial_delay" and "jobs.registry.retry.max_delay", which respectively control initial delay and maximum delay between resumptions. Fixes #44594 Fixes #65080 68212: colexec: add optimized versions of aggregate window functions r=DrewKimball a=DrewKimball **colexecwindow: add sliding window functionality to window framer** This commit adds a method `slidingWindowIntervals` to `windowFramer` operators that returns a set of `toAdd` intervals and a set of `toRemove` intervals, which indicate the rows that should be added to the current aggregation and those that should be removed, respectively. This will be used to implement the sliding window optimization for aggregate window functions such as `sum`. **colexecwindow: implement sliding window aggregator** This commit supplies a new operator, `slidingWindowAggregator`, which is used for any window aggregate functions that implement the `slidingWindowAggregateFunc` interface. Rather than aggregating over the entire window frame for each row, the `slidingWindowAggregator` operator aggregates over the rows that are in the current window frame but were not in the previous, and removes from the aggregation the rows that were in the previous window frame but not the current. This allows window aggregate functions to be evaluated in linear rather than quadratic time. **colexec: implement sliding window optimization for sum window function** This commit modifies the `sum` aggregate window function to implement the `slidingWindowAggregateFunc`, which allows it to be used in a sliding window context. This yields linear rather than quadratic scaling in the worst case, and allows the vectorized engine to meet or exceed parity with the row engine for `sum` window functions. **colexec: implement sliding window optimization for count window function** This commit modifies the count aggregate operator to implement the `slidingWindowAggregateFunc` interface so that it can be used with the sliding window optimization. **colexec: implement sliding window optimization for average window function** This commit modifies the `average` aggregate operator to implement the `slidingWindowAggregateFunc` interface so that it can be used with the sliding window optimization. **colexec: optimize count_rows window function** This commit implements an optimized version of `count_rows` that calculates the size of the window frame as soon as the window frame is calculated. This means that most of the overhead for `count_rows` now comes from calculating the window frame, which is worst-case linear time (previously, the step to retrieve the size of the frame was quadratic, though with a small constant). **colexec: optimize min and max window functions with default exclusion** This commit modifies the 'min' and 'max' aggregate window functions to implement the `slidingWindowAggregateFunc` interface, which allows them to be used in a sliding window context. However, this is only usable when the window frame never shrinks - e.g. it always contains all rows from the previous frame. This commit also provides implementations of `min` and `max` for use when the window frame can shrink. The indices of the 'next best' minimum or maximum values are stored in a priority queue that is updated for each row. Using the priority queue allows the `min` and `max` operators to avoid fully aggregating over the window frame even when the previous best value goes out of scope. Note that this implementation currently does not handle the case of non-default exclusion clause, in which case we must fall back to the quadratic approach. Fixes: #37039 Release note (performance improvement): The vectorized engine can now use the sliding-window approach to execute common aggregate functions as window functions. This allows aggregate window functions to be evaluated in linear rather than quadratic time. Currently, sum, count, average, min, and max are executed using this approach. 68433: sql: implemented placement restricted syntax for domiciling r=pawalt a=pawalt This PR combines the existing restricted placement zone config logic with the stubbed syntax to create an end-to-end PLACEMENT RESTRICTED implementation. Release note: None Note that the cluster setting for domiciling and telemetry will be added in a later PR. 68818: changefeedccl: mark avro format as no longer experimental r=[miretskiy,spiffyeng] a=HonoreDB The avro format for changefeeds now supports all column types and has been in production use for several releases. We'll now allow format=avro rather than format=experimental_avro The old string will remain supported because job payloads can persist across upgrades and downgrades. Release note (enterprise change): changefeed avro format no longer marked experimental Co-authored-by: Sajjad Rizvi <sajjad@cockroachlabs.com> Co-authored-by: Drew Kimball <drewk@cockroachlabs.com> Co-authored-by: Peyton Walters <peyton.walters@cockroachlabs.com> Co-authored-by: Aaron Zinger <zinger@cockroachlabs.com>
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build succeeded: |
The avro format for changefeeds now supports all column types
and has been in production use for several releases.
We'll now allow format=avro rather than format=experimental_avro
The old string will remain supported because job payloads can
persist across upgrades and downgrades.
Release note (enterprise change): changefeed avro format no longer marked experimental