Skip to content

changefeedccl: mark avro format as no longer experimental#68818

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
HonoreDB:deprecate_experimental
Aug 14, 2021
Merged

changefeedccl: mark avro format as no longer experimental#68818
craig[bot] merged 1 commit intocockroachdb:masterfrom
HonoreDB:deprecate_experimental

Conversation

@HonoreDB
Copy link
Copy Markdown
Contributor

The avro format for changefeeds now supports all column types
and has been in production use for several releases.
We'll now allow format=avro rather than format=experimental_avro
The old string will remain supported because job payloads can
persist across upgrades and downgrades.

Release note (enterprise change): changefeed avro format no longer marked experimental

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@HonoreDB HonoreDB marked this pull request as draft August 12, 2021 19:11
@HonoreDB HonoreDB force-pushed the deprecate_experimental branch from 62b2447 to 514ad6a Compare August 12, 2021 19:53
@HonoreDB HonoreDB marked this pull request as ready for review August 12, 2021 19:53
@HonoreDB HonoreDB force-pushed the deprecate_experimental branch 2 times, most recently from 0875f46 to 771af3a Compare August 12, 2021 21:52
@HonoreDB HonoreDB force-pushed the deprecate_experimental branch from 771af3a to 2872f8d Compare August 13, 2021 01:06
Copy link
Copy Markdown
Contributor

@miretskiy miretskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @HonoreDB, @spiffyyeng, and @stevendanna)


pkg/ccl/changefeedccl/changefeed_stmt.go, line 159 at r1 (raw file):

		if opts[changefeedbase.OptFormat] == changefeedbase.DeprecatedOptFormatAvro {
			//TODO(zinger): See if there's a way to make this work with sinkless

i guess no... i'd just drop this todo...


pkg/ccl/changefeedccl/helpers_test.go, line 654 at r1 (raw file):

// a lot of cdc test framework, use with caution. Driver-agnostic tools don't
// have clean ways of inspecting incoming notices.
func expectNotice(t *testing.T, s serverutils.TestServerInterface, sql string, expected string) {

i won't mind if this very minor detail was left outside of tests...
Like: try it manually; see if you get the notice and don't blow anything up... I'm not sure i'd necessarily test for that.

@HonoreDB HonoreDB force-pushed the deprecate_experimental branch 2 times, most recently from a2d0f77 to 506ef41 Compare August 13, 2021 15:02
@HonoreDB
Copy link
Copy Markdown
Contributor Author

bors r=[miretskiy, spiffyeng]

Copy link
Copy Markdown
Contributor Author

@HonoreDB HonoreDB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reviews!

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @miretskiy and @stevendanna)


pkg/ccl/changefeedccl/changefeed_stmt.go, line 159 at r1 (raw file):

Previously, miretskiy (Yevgeniy Miretskiy) wrote…

i guess no... i'd just drop this todo...

Ah well. Done.


pkg/ccl/changefeedccl/helpers_test.go, line 654 at r1 (raw file):

Previously, miretskiy (Yevgeniy Miretskiy) wrote…

i won't mind if this very minor detail was left outside of tests...
Like: try it manually; see if you get the notice and don't blow anything up... I'm not sure i'd necessarily test for that.

Yeah, testing this was one of those things where I thought it'd be easy and it ended up taking way too long and being awkward but at that point I was emotionally committed. I usually just grumble and delete the code when that happens but eh, it's a cheap test.

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 13, 2021

Build failed:

The avro format for changefeeds now supports all column types
and has been in production use for several releases.
We'll now allow format=avro rather than format=experimental_avro
The old string will remain supported because job payloads can
persist across upgrades and downgrades.

Release note (enterprise change): changefeed avro format no longer marked experimental
@HonoreDB HonoreDB force-pushed the deprecate_experimental branch from 506ef41 to 69e70e7 Compare August 13, 2021 17:07
@HonoreDB
Copy link
Copy Markdown
Contributor Author

bors retry

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 13, 2021

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 13, 2021

This PR was included in a batch that was canceled, it will be automatically retried

craig bot pushed a commit that referenced this pull request Aug 14, 2021
66889: jobs: retry jobs with exponential backoff r=ajwerner a=sajjadrizvi

This commit adds a mechanism to retry jobs with exponentially increasing
delays. This is achieved through two new columns in system.jobs table,
last_run and num_runs. In addition, this commit adds cluster settings
to control exponential-backoff parameters, initial delay and max delay,
with corresponding settings `jobs.registry.retry.initial_delay` and
`jobs.registry.retry.max_delay`. Finally, this commit adds a new
partial-index in the jobs table that improves the performance of periodic 
queries run by registry in each node.

Release note (general change): The behavior for retrying jobs, which fail
due to a retriable error or due to job coordinator failure, is now delayed
using exponential backoff. Before this change, jobs which failed in a
retryable manner, would be resumed immediately on a different coordinator.
This change reduces the impact of recurrently failing jobs on the cluster.
This change adds two new cluster settings that control this behavior:
"jobs.registry.retry.initial_delay" and "jobs.registry.retry.max_delay",
which respectively control initial delay and maximum delay between 
resumptions.

Fixes #44594
Fixes #65080

68212: colexec: add optimized versions of aggregate window functions r=DrewKimball a=DrewKimball

**colexecwindow: add sliding window functionality to window framer**

This commit adds a method `slidingWindowIntervals` to `windowFramer`
operators that returns a set of `toAdd` intervals and a set of
`toRemove` intervals, which indicate the rows that should be added
to the current aggregation and those that should be removed, respectively.
This will be used to implement the sliding window optimization for
aggregate window functions such as `sum`.

**colexecwindow: implement sliding window aggregator**

This commit supplies a new operator, `slidingWindowAggregator`, which
is used for any window aggregate functions that implement the
`slidingWindowAggregateFunc` interface. Rather than aggregating over
the entire window frame for each row, the `slidingWindowAggregator`
operator aggregates over the rows that are in the current window
frame but were not in the previous, and removes from the aggregation
the rows that were in the previous window frame but not the current.
This allows window aggregate functions to be evaluated in linear rather
than quadratic time.

**colexec: implement sliding window optimization for sum window function**

This commit modifies the `sum` aggregate window function to implement
the `slidingWindowAggregateFunc`, which allows it to be used in a
sliding window context. This yields linear rather than quadratic scaling
in the worst case, and allows the vectorized engine to meet or exceed
parity with the row engine for `sum` window functions.

**colexec: implement sliding window optimization for count window function**

This commit modifies the count aggregate operator to implement the
`slidingWindowAggregateFunc` interface so that it can be used with
the sliding window optimization.

**colexec: implement sliding window optimization for average window function**

This commit modifies the `average` aggregate operator to implement the
`slidingWindowAggregateFunc` interface so that it can be used with the
sliding window optimization.

**colexec: optimize count_rows window function**

This commit implements an optimized version of `count_rows` that
calculates the size of the window frame as soon as the window frame
is calculated. This means that most of the overhead for `count_rows`
now comes from calculating the window frame, which is worst-case
linear time (previously, the step to retrieve the size of the frame
was quadratic, though with a small constant).

**colexec: optimize min and max window functions with default exclusion**

This commit modifies the 'min' and 'max' aggregate window functions
to implement the `slidingWindowAggregateFunc` interface, which allows
them to be used in a sliding window context. However, this is only
usable when the window frame never shrinks - e.g. it always contains
all rows from the previous frame.

This commit also provides implementations of `min` and `max` for use
when the window frame can shrink. The indices of the 'next best'
minimum or maximum values are stored in a priority queue that is
updated for each row. Using the priority queue allows the `min` and
`max` operators to avoid fully aggregating over the window frame
even when the previous best value goes out of scope. Note that this
implementation currently does not handle the case of non-default
exclusion clause, in which case we must fall back to the quadratic
approach.

Fixes: #37039

Release note (performance improvement): The vectorized engine can now
use the sliding-window approach to execute common aggregate functions 
as window functions. This allows aggregate window functions to be evaluated
in linear rather than quadratic time. Currently, sum, count, average, min, and 
max are executed using this approach.

68433: sql: implemented placement restricted syntax for domiciling r=pawalt a=pawalt

This PR combines the existing restricted placement zone config logic
with the stubbed syntax to create an end-to-end PLACEMENT RESTRICTED
implementation.

Release note: None

Note that the cluster setting for domiciling and telemetry will be added in a later PR.

68818: changefeedccl: mark avro format as no longer experimental r=[miretskiy,spiffyeng] a=HonoreDB

The avro format for changefeeds now supports all column types
and has been in production use for several releases.
We'll now allow format=avro rather than format=experimental_avro
The old string will remain supported because job payloads can
persist across upgrades and downgrades.

Release note (enterprise change): changefeed avro format no longer marked experimental

Co-authored-by: Sajjad Rizvi <sajjad@cockroachlabs.com>
Co-authored-by: Drew Kimball <drewk@cockroachlabs.com>
Co-authored-by: Peyton Walters <peyton.walters@cockroachlabs.com>
Co-authored-by: Aaron Zinger <zinger@cockroachlabs.com>
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 14, 2021

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 14, 2021

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 14, 2021

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants