Skip to content

VReplication support for non-PRIMARY KEY (Online DDL context): alternate PR#8420

Closed
shlomi-noach wants to merge 54 commits intovitessio:mainfrom
planetscale:onlineddl-vrepl-source-target-columns
Closed

VReplication support for non-PRIMARY KEY (Online DDL context): alternate PR#8420
shlomi-noach wants to merge 54 commits intovitessio:mainfrom
planetscale:onlineddl-vrepl-source-target-columns

Conversation

@shlomi-noach
Copy link
Copy Markdown
Contributor

@shlomi-noach shlomi-noach commented Jul 6, 2021 •
edited
Loading

Description

A take/twist on #8364, per comment

Mostly exactly the same as #8364 , except:

  • Rule's proto has source_unique_key_columns instead of source_unique_key, and target_unique_key_columns instead of target_unique_key.
  • rowstreamer doesn't need to inspect information_schema.columns, since it relies on source_unique_key_columns
  • vreplicator likewise doesn't need to inspect information_schema.columns, since it relies on target_unique_key_columns

Related Issue(s)

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

@shlomi-noach
refactor: pass colInfos instead of colInfoMap. Add comments as needed
671c8d6 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Adding UniqueKey to Rule
686a3de 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
adapt to replication_plan_builder refactor
3e2c740 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
convenience method: return comment string, unquote if needed
b0ade0a 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Table.FindColumn
2877b55 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
rowstreamer: read unique_key hint from comment dircetives, evaluate '…
0a57edd 
…PK' columns from unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
unique key analysis query
c6814c4 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
when rule indicates non-PRIMARY UniqueKey, override PK ColInfo with u…
77e593c 
…nique key specific info

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
generated code
61362c1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
onlineddl/vrepl: read source/target unique keys, evaluate best sharee…
9cbfa61 
…d unique key

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
proto: souce_unique_key and target_unique_key
dfcd76a 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
proto: SourceUniqueKey and TargetUniqueKey
0490612 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
BaseShowTableUniqueKey query
0fb881c 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
source unique key and target unique key split
a04a5b4 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
sourceUniqueKeys and targetUniqueKeys don't need to be fields in VRepl
9560db7 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
adding unit test
4ceae54 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
more unit test cases
6ee5d48 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
enforce use of mapping if exists
84947cb 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
vreplicator: refactor logic, add unit test
a7339d1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
code comment
3981575 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
vrepl_mini_stress_test: double the tests, use a non PK unique key wit…
f62c1dc 
…h opposite ordering

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Merge branch 'main' into online-ddl-vrpel-suite-pk-uk
dde35e0 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Merge branch 'main' into online-ddl-vrpel-suite-pk-uk
53184b7 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Adding test files
b5f0aa4 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
restore vrepl_stress test to original (main). Will create a new vrepl…
56ed681 
…_stress_suite test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
new endtoend test: vrepl_stress_suite, testing a myriad of schema cha…
f445dee 
…nges under stress

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
new endtoend test: vrepl_stress_suite, testing a myriad of schema cha…
20ab272 
…nges under stress

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
stricter data validation; more test cases
14056e0 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactor all onlineddl tests: reduce wait time by polling migration s…
9f433c1 
…tatus

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactor all onlineddl tests: reduce wait time by polling migration s…
1516234 
…tatus

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
stricter order of selcet for data comparison
c0f3653 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
test suite: use small packet size to ensure we use lastPK logic on vc…
e5a1787 
…opier side

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Merge branch 'main' into online-ddl-vrpel-suite-pk-uk
7472ed9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
more concurrency
61fa980 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
tracking op_order fo rbetter failure analysis. adding a couple tests.…
66d4255 
… Increasing contention/stress

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactor sleep/concurrency values
b5b09e9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
incorporated fix from vitessio#8390
b76e4c1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
provide alias for textual columns with convert() function, so that fi…
7ac7a8e 
…eld name still reflects column name

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
variable rename
6646e10 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
only apply convert() function if either source or target charsets are…
093d32f 
… non trivial (no need for convert if both are trivial)

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Remove debug comments
00280ef 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Merge branch 'main' into online-ddl-vrpel-suite-pk-uk
ba72d0e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
one more test
31271f7 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
increase timeout for vrepl tests
a2d09e0 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
more relaxed timeouts
b54f5b1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
less load. Current load mostly causes timeouts, which is not what we'…
67d1e8d 
…re looking for in tests

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
merge master, resolve conflict
940c433 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Merge branch 'main' into online-ddl-vrpel-suite-pk-uk
9f874c8 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
go 1.16
6f13166 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
source_unique_key => source_unique_key_columns, target_unique_key => …
01c0abf 
…target_unique_key_columns, values populated by vrepl and no need for information_schema.columns queries on source and target

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach mentioned this pull request Jul 6, 2021
Merged
2 tasks
@shlomi-noach shlomi-noach added Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature) labels Jul 6, 2021
@shlomi-noach
source_unique_key_columns and target_unique_key_columns are now simpl…
df0d002 
…e strings instead of string arrays. The value of source_unique_key_columns is comma delimited (and escaped) list of column names; similarly for target_unique_key_columns

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach force-pushed the onlineddl-vrepl-source-target-columns branch from c3b7eae to df0d002 Compare July 7, 2021 06:12
@shlomi-noach
reject unique keys leading with float/double data type column
60b7c15 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach mentioned this pull request Jul 7, 2021
Merged
2 tasks
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Jul 12, 2021

This is now merged into #8364

@shlomi-noach shlomi-noach deleted the onlineddl-vrepl-source-target-columns branch August 4, 2021 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@frouioui frouioui Awaiting requested review from frouioui frouioui will be requested when the pull request is marked ready for review frouioui is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps will be requested when the pull request is marked ready for review rohit-nayak-ps is a code owner

@mattlord mattlord Awaiting requested review from mattlord mattlord will be requested when the pull request is marked ready for review mattlord is a code owner

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal will be requested when the pull request is marked ready for review harshit-gangal is a code owner

@systay systay Awaiting requested review from systay systay will be requested when the pull request is marked ready for review systay is a code owner

@timvaillancourt timvaillancourt Awaiting requested review from timvaillancourt timvaillancourt will be requested when the pull request is marked ready for review timvaillancourt is a code owner

@beingnoble03 beingnoble03 Awaiting requested review from beingnoble03 beingnoble03 will be requested when the pull request is marked ready for review beingnoble03 is a code owner

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shlomi-noach