VReplication Copy Phase: Parallelize Bulk Inserts#10828
Merged
mattlord merged 2 commits intovitessio:mainfrom Oct 26, 2022
Merged
VReplication Copy Phase: Parallelize Bulk Inserts#10828mattlord merged 2 commits intovitessio:mainfrom
mattlord merged 2 commits intovitessio:mainfrom
Conversation
Contributor
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Bug fixes
Non-trivial changes
New/Existing features
Backward compatibility
|
8 tasks
Collaborator
|
With @mattlord help I have been benchmarking the performance of this PR alongside #10788. Both PRs improve performance between 10-200%, depending on factors like CPU load on the data source. Because this PR came first, already has undergone some code review, and is simpler than #10788, I'm going to add more work to this PR.
|
8d13a17 to
b6d6298
Compare
a877a99 to
a7da4da
Compare
2a41fd9 to
9d90fe6
Compare
ce597c0 to
1a7bbb5
Compare
4facf7b to
fb29459
Compare
maxenglander
reviewed
Sep 8, 2022
maxenglander
reviewed
Sep 8, 2022
maxenglander
reviewed
Sep 8, 2022
maxenglander
reviewed
Sep 8, 2022
mattlord
reviewed
Sep 9, 2022
mattlord
reviewed
Sep 11, 2022
rohit-nayak-ps
commented
Oct 9, 2022
3aea5b5 to
5d8bde8
Compare
Collaborator
|
Have squashed all the commits. Was too annoying to try to fix the commit that was missing DCO with rebase. Previous commit history preserved over here just in case. |
…ovements
This PR adds a new experimental flag
`--vreplication-parallel-insert-workers=[integer]` that enables parallel bulk
inserts during the copy phase of VReplication. It is not set by default.
This PR also adds three new VReplication stats:
- `VReplicationTableCopyTimings` is similar to `PhaseTimings{Copy}`, but
provides per-table-per-stream observability and updates throughout the copy
phase, instead of at the end of the copy phase.
- `VReplicationTableCopyRowCounts` counts rows copied per-table-per-stream
throughout the copy phase.
- `VReplicationBulkInsertConcurrency` gauges the configured number of concurrent
insertion workers.
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
5d8bde8 to
41a4233
Compare
mattlord
reviewed
Oct 25, 2022
Member
mattlord
left a comment
There was a problem hiding this comment.
Once again, great work on this! I only had some minor nits/questions/suggestions. We can huddle on those and get this merged. ❤️
Comment on lines
+166
to
+167
| bps.TableCopyRowCounts = stats.NewCountersWithSingleLabel("", "", "Table", "") | ||
| bps.TableCopyTimings = stats.NewTimings("", "", "Table") |
Member
There was a problem hiding this comment.
Do you have a sample of what this looks like in the /debug/vars output?
Collaborator
There was a problem hiding this comment.
Here's a sample:
"VReplicationTableCopyRowCounts": {"src.-.copy.1.data": 8388608},
| func (vr *vreplicator) readSettings(ctx context.Context) (settings binlogplayer.VRSettings, numTablesToCopy int64, err error) { | ||
| settings, err = binlogplayer.ReadVRSettings(vr.dbClient, vr.id) | ||
| // Same as readSettings, but stores some of the results on this vr. | ||
| func (vr *vreplicator) loadSettings(ctx context.Context, dbClient *vdbClient) (settings binlogplayer.VRSettings, numTablesToCopy int64, err error) { |
Member
There was a problem hiding this comment.
I don't see the need for these two function changes, what am I missing?
Collaborator
There was a problem hiding this comment.
I want to be able to re-use logic in vreplicator.go for setting things like setting and unsettingsql_mode and disabling FK checks. I want to use that logic in each new parallel MySQL conn that's created in vcopier.go. The current vreplicator.go code makes that a little difficult because it changes state in vreplicator.
To address that I move the logic I wanted to re-use into a stateless fn readSettings, and move the stateful stuff into loadSettings.
Signed-off-by: Max Englander <max@planetscale.com>
mattlord
approved these changes
Oct 26, 2022
This was referenced Feb 13, 2023
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new experimental
vttabletflag--vreplication-parallel-insert-workers=[integer]that enables parallel bulk inserts during the copy phase of VReplication. It is disabled (1) by default.This PR also adds new VReplication stats:
VReplicationTableCopyTimings: similar toPhaseTimings{Copy}, but provides per-table-per-stream observability and updates throughout the copy phase, instead of at the end of the copy phase.VReplicationTableCopyRowCounts: counts rows copied per-table-per-stream throughout the copy phase.Approach
When parallel inserts are not enabled, VReplication behaves the same way as before this PR, with the exception that the new stats are used.
When setting
--vreplication-parallel-insert-workers> 1 the bulk inserts — eachexecuted as a single transaction from the vstream packet contents — may happen in parallel and
out-of-order, but the commits of those transactions are still serialized in order.
Other aspects of the VReplication copy-phase logic are preserved:
_vt.copy_statealways follow their corresponding inserts from within the vstream packet.commitfor the vstream packet always follows the corresponding write to_vt.copy_state._vt.copy_stateandcommitsteps (steps 2 and 3 above) for PK1 will both precede the_vt.copy_statewrite and commit steps of PK2.Other phases (catchup, fast-forward, "running") are not changed by this PR.
Checklist
Impacted Areas in Vitess
Components that this PR will affect: