Skip to content

colbuilder: disable wrapping of changefeed processors#55616

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
yuzefovich:changefeeds-vec
Oct 20, 2020
Merged

colbuilder: disable wrapping of changefeed processors#55616
craig[bot] merged 1 commit intocockroachdb:masterfrom
yuzefovich:changefeeds-vec

Conversation

@yuzefovich
Copy link
Copy Markdown
Member

@yuzefovich yuzefovich commented Oct 15, 2020 •
edited
Loading

The root of the problem is that Columnarizer has buffering behavior -
in 20.1 it will be hanging until coldata.BatchSize() (1024 by default)
rows are emitted by the changefeed. On 20.2 and master due to dynamic
batch size behavior it will still be hanging but in a slightly
different manner.

This is less of a problem on 20.2 and the current master because the
vectorized engine will not be used for the changefeed DistSQL flow since
the vectorized row count threshold is never met for it (the estimated
row count for the plan is 0, so unless a user does
SET vectorize_row_count_threshold=0; or
SET vectorize=experimental_always;, we will always use row-by-row
engine). In 20.1 the meaning of vectorize=on was different - we never
looked at the threshold and used the vectorized engine if it was
supported.

In order to fix this issue we simply refuse to wrap the changefeed
processors, so the row-by-row engine will be always used for changefeed
flows.

Fixes: #55605.

Release note (bug fix): The current implementation of changefeeds is
incompatible with the vectorized engine, so whenever the latter is being
used to run the former, it could hang indefinitely. This is now fixed.
Namely, on 20.2 releases this could happen if the user runs
SET vectorize_row_count_threshold=0;, and on 20.1 releases - if the
user runs SET vectorize=on.

@yuzefovich yuzefovich requested review from a team and asubiotto October 15, 2020 20:47
@cockroach-teamcity
Copy link
Copy Markdown
Member

cockroach-teamcity commented Oct 15, 2020

This change is Reviewable

@yuzefovich
Copy link
Copy Markdown
Member Author

yuzefovich commented Oct 15, 2020

I think this PR fixes the issue for the time being because there is no benefit in planning a vectorized flow for changefeeds (we're simply adding a pair of Columnarizer - Materializer on top of ChangeFrontier) and I doubt we'll be vectorizing the changefeed processors any time soon (if ever).

@yuzefovich
Copy link
Copy Markdown
Member Author

yuzefovich commented Oct 15, 2020

Interestingly, I prototyped making a columnarizer to have two modes of operation (either "buffering" or "streaming") here, and I hit an issue with clearing up memory monitoring infra (similar to #55408) which appears to be some kind of a race (a guess based on a quick debugging session). I think it's worth getting to the bottom of that.

asubiotto
asubiotto reviewed Oct 19, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@asubiotto asubiotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating. Can you add a test or modify an existing test that would fail with the existing behavior?

This is less of a problem on 20.2 and the current master because the
vectorized engine will not be used for the changefeed DistSQL flow since
the vectorized row count threshold is never met for it (the estimated
row count for the plan is 0

Is this always true?

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @asubiotto)

@asubiotto
Copy link
Copy Markdown
Contributor

asubiotto commented Oct 19, 2020

This likely needs a backport for all versions that have a vectorized setting

@yuzefovich yuzefovich requested review from a team and miretskiy and removed request for a team October 19, 2020 15:59
yuzefovich
yuzefovich commented Oct 19, 2020
View reviewed changes
Copy link
Copy Markdown
Member Author

@yuzefovich yuzefovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a test.

Is this always true?

Yes, because we create a physical plan manually in

cockroach/pkg/ccl/changefeedccl/changefeed_dist.go

Lines 160 to 170 in 3e3aaf3

p := sql.MakePhysicalPlan(gatewayNodeID)
p.AddNoInputStage(corePlacement, execinfrapb.PostProcessSpec{}, changefeedResultTypes, execinfrapb.Ordering{})
p.AddSingleGroupStage(
gatewayNodeID,
execinfrapb.ProcessorCoreUnion{ChangeFrontier: &changeFrontierSpec},
execinfrapb.PostProcessSpec{},
changefeedResultTypes,
)
p.PlanToStreamColMap = []int{1, 2, 3}
dsp.FinalizePlan(planCtx, &p)
and we don't update MaxEstimatedRowCount field before running the plan, so it always remains at 0.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @miretskiy)

@yuzefovich
colbuilder: disable wrapping of changefeed processors
84ce8e4 
The root of the problem is that Columnarizer has buffering behavior -
in 20.1 it will be hanging until coldata.BatchSize() (1024 by default)
rows are emitted by the changefeed. On 20.2 and master due to dynamic
batch size behavior it will still be hanging but in a slightly
different manner.

This is less of a problem on 20.2 and the current master because the
vectorized engine will not be used for the changefeed DistSQL flow since
the vectorized row count threshold is never met for it (the estimated
row count for the plan is 0, so unless a user does
`SET vectorize_row_count_threshold=0;` or
`SET vectorize=experimental_always;`, we will always use row-by-row
engine). In 20.1 the meaning of `vectorize=on` was different - we never
looked at the threshold and used the vectorized engine if it was
supported.

In order to fix this issue we simply refuse to wrap the changefeed
processors, so the row-by-row engine will be always used for changefeed
flows.

Release note (bug fix): The current implementation of changefeeds is
incompatible with the vectorized engine, so whenever the latter is being
used to run the former, it could hang indefinitely. This is now fixed.
Namely, on 20.2 releases this could happen if the user runs
`SET vectorize_row_count_threshold=0;`, and on 20.1 releases - if the
user runs `SET vectorize=on`.
@yuzefovich yuzefovich mentioned this pull request Oct 19, 2020
Closed
asubiotto
asubiotto approved these changes Oct 20, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@asubiotto asubiotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r1.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @miretskiy)

@yuzefovich yuzefovich removed the request for review from miretskiy October 20, 2020 14:39
@yuzefovich
Copy link
Copy Markdown
Member Author

yuzefovich commented Oct 20, 2020

TFTR!

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Oct 20, 2020

Build succeeded:

@craig craig bot merged commit e16811f into cockroachdb:master Oct 20, 2020
@yuzefovich yuzefovich deleted the changefeeds-vec branch October 20, 2020 16:16
This was referenced Oct 20, 2020
Merged
Merged
@yuzefovich yuzefovich mentioned this pull request Dec 1, 2020
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@asubiotto asubiotto asubiotto approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

cdc,sql: core changefeeds don't work as expected with vectorized=on

3 participants

@yuzefovich @cockroach-teamcity @asubiotto