VDiff2: Support Resuming VDiffs by mattlord · Pull Request #10497 · vitessio/vitess

mattlord · 2022-06-13T22:48:25Z

Description

This work allows you to explicitly resume an existing VDiff workflow using the new resume <UUID> VDiff action. VDiff will then resume, picking up where it left off and comparing the records where the Primary Key column(s) are greater than the last record processed — with the progress and other status information saved when the run ends. This allows you to:

Resume a VDiff that may have encountered an ephemeral error (adding an auto-retry on error will come later)
Do approximate rolling or differential VDiffs (e.g. done after MoveTables finishes the initial copy phase and then again just before SwitchTraffic)

In order to better support this for users and testing, we also now show the CompletedAt timestamp in the VDiff summary, when it does complete w/o error, and we also show the RowsCompared during the run to accurately reflect the work done on resuming.

⚠️ We cannot guarantee accurate results for resume across major MySQL versions due to implied collation differences. For example: in MySQL 5.7 a table using the utf8mb4 character set will use the utf8mb4_general_ci collation by default whereas in MySQL 8.0 a table using the utf8mb4 character set will use the utf8mb4_0900_ai_ci collation by default. These two collations don't produce the same ordering and thus when resuming from the last PK we processed in the previous VDiff run, e.g. using:
select c_uuid, created_at, dstuff, dtstuff, dbstuff, cstuff from db_order_test where (c_uuid = 'b169-da8df852-9bad-22000b029685-14e3' and created_at > '2020-05-06 07:22:17') or (c_uuid > 'b169-da8df852-9bad-22000b029685-14e3') order by c_uuid, created_at
The results will not be the same. And because we're not diffing the full table, the reconciliation step we take at the end does not help us here as in some cases we will have processed a row on one side only (rather than processing it on both sides but in different orders). The only way to ensure we get the same results on both sides in these cases would be to negotiate the lowest common denominator for the character set between versions — in this case utf8mb4_general_ci — and explicitly pass that down on both sides like this:
select c_uuid, created_at, dstuff, dtstuff, dbstuff, cstuff from db_order_test where (c_uuid = 'b169-da8df852-9bad-22000b029685-14e3' and created_at > '2020-05-06 07:22:17') or (c_uuid > 'b169-da8df852-9bad-22000b029685-14e3') order by c_uuid, created_at collate utf8mb4_general_ci
But that will have a catastrophic impact on performance and overhead as when the collation specified in the query does not match the collation used for the PK index, a filesort will be needed to sort the entire resultset before streaming any rows. For this reason, we will document the fact that resume is not guaranteed to produce accurate results when the source and target tables are not using the same collation.

Click here for an example manual test and example results

## Begin Test
pushd examples/local

./401_teardown.sh; ./101_initial_cluster.sh; ./201_customer_tablets.sh; ./202_move_tables.sh && echo -e "\n---------------------\n\n"

mysql -h 127.0.0.1 -P 15306 -e 'insert into customer values (1, "mlord@planetscale.com"), (2, "mlord@planetscale.com"), (3, "mlord@planetscale.com"), (4, "mlord@planetscale.com")'

mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock -e "analyze table vt_customer.customer" >/dev/null

for id in {0..2}; do mysql -u root --socket=/opt/vtdataroot/vt_000000010${id}/mysql.sock -e "set global general_log=ON"; done; for id in {0..2}; do mysql -u root --socket=/opt/vtdataroot/vt_000000020${id}/mysql.sock -e "set global general_log=ON"; done

vtctlclient VDiff -- --v2 --format json --limit 2 customer.commerce2customer create 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" && grep -i select /opt/vtdataroot/*/data/pslord.log | grep -i "order by customer_id" && echo -e "\n---------------------\n\n"

vtctlclient VDiff -- --v2 --format json --limit 3 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" && grep -i select /opt/vtdataroot/*/data/pslord.log | grep -i "order by customer_id" && echo -e "\n---------------------\n\n"

vtctlclient VDiff -- --v2 --format json --limit 4 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" && grep -i select /opt/vtdataroot/*/data/pslord.log | grep -i "order by customer_id" && echo -e "\n---------------------\n\n"

vtctlclient VDiff -- --v2 --format json --limit 4 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" && grep -i select /opt/vtdataroot/*/data/pslord.log | grep -i "order by customer_id" && echo -e "\n---------------------\n\n"

mysql -h 127.0.0.1 -P 15306 -e 'insert into customer values (5, "mlord@planetscale.com"), (6, "mlord@planetscale.com"), (7, "mlord@planetscale.com"), (8, "mlord@planetscale.com")'

mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock -e "analyze table vt_customer.customer" >/dev/null

vtctlclient VDiff -- --v2 --format json customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" &&  grep -i select /opt/vtdataroot/*/data/pslord.log | grep -i "order by customer_id" && echo -e "\n---------------------\n\n"

vtctlclient VDiff -- --v2 --format json customer.commerce2customer show 4c664dc2-eba9-11ec-9ef7-920702940ee0

mysql -h 127.0.0.1 -P 15306 -e 'insert into customer values (20, "mlord@planetscale.com"), (21, "mlord@planetscale.com")'

mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock -e "analyze table vt_customer.customer" >/dev/null

mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock -e 'update vt_customer.customer set email="mlord-updated@planetscale.com" where customer_id=20'

vtctlclient VDiff -- --v2 --format json customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0 && sleep 5 && mysql -u root --socket=/opt/vtdataroot/vt_0000000200/mysql.sock --binary-as-hex=false -e "select * from _vt.vdiff_table where table_name='customer'\G" && echo -e "\n---------------------\n\n"

vtctlclient VDiff -- --v2 --format json customer.commerce2customer show 4c664dc2-eba9-11ec-9ef7-920702940ee0

popd
## End Test


....


## Start Results
Waiting for workflow to start:

Workflow started successfully with 1 stream(s)

Following vreplication streams are running for workflow customer.commerce2customer:

id=1 on 0/zone1-0000000200: Status: Copying. VStream Lag: 0s.


---------------------


{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:1 values:"1"}
   table_rows: 4
rows_compared: 1
     mismatch: 0
       report: {"TableName": "customer", "MatchingRows": 1, "ProcessedRows": 1, "MismatchedRows": 0, "ExtraRowsSource": 0, "ExtraRowsTarget": 0}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:26
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:26.525291Z	   51 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:26.594029Z	   42 Query	select customer_id, email from customer order by customer_id

---------------------


{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:1 values:"3"}
   table_rows: 4
rows_compared: 2
     mismatch: 0
       report: {"TableName": "customer", "MatchingRows": 2, "ProcessedRows": 2, "MismatchedRows": 0, "ExtraRowsSource": 0, "ExtraRowsTarget": 0}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:31
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:26.525291Z	   51 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:31.462618Z	   61 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:26.594029Z	   42 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:31.528514Z	   54 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id

---------------------


{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:1 values:"4"}
   table_rows: 4
rows_compared: 1
     mismatch: 0
       report: {"TableName": "customer", "MatchingRows": 1, "ProcessedRows": 1, "MismatchedRows": 0, "ExtraRowsSource": 0, "ExtraRowsTarget": 0}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:36
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:26.525291Z	   51 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:31.462618Z	   61 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:36.678800Z	   71 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:26.594029Z	   42 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:31.528514Z	   54 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:36.726510Z	   66 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id

---------------------


{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:1 values:"4"}
   table_rows: 4
rows_compared: 0
     mismatch: 0
       report: {"TableName": "customer", "MatchingRows": 0, "ProcessedRows": 0, "MismatchedRows": 0, "ExtraRowsSource": 0, "ExtraRowsTarget": 0}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:41
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:26.525291Z	   51 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:31.462618Z	   61 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:36.678800Z	   71 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:41.659760Z	   81 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:26.594029Z	   42 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:31.528514Z	   54 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:36.726510Z	   66 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:41.754246Z	   78 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id

---------------------


{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:1 values:"8"}
   table_rows: 8
rows_compared: 4
     mismatch: 0
       report: {"TableName": "customer", "MatchingRows": 4, "ProcessedRows": 4, "MismatchedRows": 0, "ExtraRowsSource": 0, "ExtraRowsTarget": 0}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:47
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:26.525291Z	   51 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:31.462618Z	   61 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:36.678800Z	   71 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:41.659760Z	   81 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id
/opt/vtdataroot/vt_0000000102/data/pslord.log:2022-06-26T22:58:46.897918Z	   91 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:26.594029Z	   42 Query	select customer_id, email from customer order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:31.528514Z	   54 Query	select customer_id, email from customer where (customer_id > 1) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:36.726510Z	   66 Query	select customer_id, email from customer where (customer_id > 3) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:41.754246Z	   78 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id
/opt/vtdataroot/vt_0000000202/data/pslord.log:2022-06-26T22:58:47.007630Z	   89 Query	select customer_id, email from customer where (customer_id > 4) order by customer_id

---------------------


{
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
	"RowsCompared": 4,
	"HasMismatch": false,
	"Shards": "0",
	"StartedAt": "2022-06-26 22:58:45",
	"CompletedAt": "2022-06-26 22:58:47"
}
{
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0"
}
*************************** 1. row ***************************
     vdiff_id: 1
   table_name: customer
        state: completed
       lastpk: fields:{name:"customer_id" type:INT64 table:"customer" org_table:"customer" database:"vt_customer" org_name:"customer_id" column_length:20 charset:63 flags:49667} rows:{lengths:2 values:"21"}
   table_rows: 10
rows_compared: 2
     mismatch: 1
       report: {"TableName": "customer", "MatchingRows": 1, "ProcessedRows": 2, "MismatchedRows": 1, "ExtraRowsSource": 0, "ExtraRowsTarget": 0, "MismatchedRowsSample": [{"Source": {"Row": {"email": "mlord@planetscale.com", "customer_id": "20"}}, "Target": {"Row": {"email": "mlord-updated@planetscale.com", "customer_id": "20"}}}]}
   created_at: 2022-06-26 18:58:26
   updated_at: 2022-06-26 18:58:51

---------------------


{
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
	"RowsCompared": 2,
	"HasMismatch": true,
	"Shards": "0",
	"StartedAt": "2022-06-26 22:58:50",
	"CompletedAt": "2022-06-26 22:58:52",
	"TableSummary": {
		"corder": {
			"TableName": "corder",
			"State": "completed",
			"RowsCompared": 0,
			"MatchingRows": 0,
			"MismatchedRows": 0,
			"ExtraRowsSource": 0,
			"ExtraRowsTarget": 0
		},
		"customer": {
			"TableName": "customer",
			"State": "completed",
			"RowsCompared": 2,
			"MatchingRows": 1,
			"MismatchedRows": 1,
			"ExtraRowsSource": 0,
			"ExtraRowsTarget": 0
		}
	},
	"Reports": {
		"corder": {
			"0": {
				"TableName": "corder",
				"ProcessedRows": 0,
				"MatchingRows": 0,
				"MismatchedRows": 0,
				"ExtraRowsSource": 0,
				"ExtraRowsTarget": 0
			}
		},
		"customer": {
			"0": {
				"TableName": "customer",
				"ProcessedRows": 2,
				"MatchingRows": 1,
				"MismatchedRows": 1,
				"ExtraRowsSource": 0,
				"ExtraRowsTarget": 0,
				"MismatchedRowsSample": [
					{
						"Source": {
							"Row": {
								"customer_id": "20",
								"email": "mlord@planetscale.com"
							}
						},
						"Target": {
							"Row": {
								"customer_id": "20",
								"email": "mlord-updated@planetscale.com"
							}
						}
					}
				]
			}
		}
	}
}
## End Results

Related Issue(s)

Fixes: Resumable VDiffs #10495

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added
Documentation: Document vdiff resume <uuid> command website#1060

And fix bug that caused the row estimate to always be 0. Signed-off-by: Matt Lord <mattalord@gmail.com>

Still need to actually resume ... Signed-off-by: Matt Lord <mattalord@gmail.com>

What's left is adding the lastPK results to the SELECT used on the source and target tablets for retrieving the rows to compare. Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

In the case we processed no new rows the lastpk should remain the same. Before this we ended up clearing the lastpk out when doing a resume that had no new rows to examine, thus causing the subsequent resume to diff all of the rows again. Signed-off-by: Matt Lord <mattalord@gmail.com>

And add invariant check in updating table status Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

Remove debugging code used during development. Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

Turns out that log does not support this verb. So we can stick to using it with fmt instead. Signed-off-by: Matt Lord <mattalord@gmail.com>

This provides accurate progress info during the resume run Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

And update release notes for new output Signed-off-by: Matt Lord <mattalord@gmail.com>

Since this was not working before this PR. :-) Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

rohit-nayak-ps · 2022-06-27T10:10:57Z

go/test/endtoend/vreplication/vdiff2_test.go

+		tables:        "customer,Lead,Lead-1",
+		resume:        true,
+		resumeInsert:  `insert into customer(cid, name, typ) values(12345678, 'Testy McTester', 'soho')`,
+		testCLIErrors: true, // test for errors in the simplest test case


Since the CLI test is an independent one and called only once, we should probably just take it out of testWorkflow() and call it as a separate sub-test.

I had initially gone that route too, but then there's no workflows to test against — as testWorkflow() does SwitchWrites and Complete at the end — and the only errors you get are a result of the fact that the workflow does not exist. :-) In doing that, I also realized there's some follow-up work to do in handling those error cases well (today you get an error about a missing key as the VDiff metadata is there but the backing vreplication workflow is not).

go/test/endtoend/vreplication/vdiff2_test.go

go/vt/vtctl/vdiff2.go

rohit-nayak-ps · 2022-06-27T11:21:57Z

go/vt/vttablet/tabletmanager/vdiff/schema.go

+										IF(vdt.mismatch = 1, 1, 0) as has_mismatch, vdt.report as report
+										from _vt.vdiff as vd inner join _vt.vdiff_table as vdt on (vd.id = vdt.vdiff_id)
+										where vdt.vdiff_id = %d`
+	// sqlUpdateVDiffState has a penultimate placeholder for any additional columns you want to update, e.g. `, foo = 1`


Signed-off-by: Matt Lord <mattalord@gmail.com>

rohit-nayak-ps

looks great to me!

And fix a number of bugs discovered related to incorrect VDiff summary handling and other more minor things. Signed-off-by: Matt Lord <mattalord@gmail.com>

* CherryPick: VDiff2: Support Resuming VDiffs (#10497) And fix a number of bugs discovered related to incorrect VDiff summary handling and other more minor things. Signed-off-by: Matt Lord <mattalord@gmail.com>

Periodically save VDiff progress per table

a3cd8a3

And fix bug that caused the row estimate to always be 0. Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication release notes (needs details) This PR needs to be listed in the release notes in a dedicated section (deprecation notice, etc...) labels Jun 14, 2022

Add resume command & action handling

77ba448

Still need to actually resume ... Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vdiff2_resume branch from 397f6d6 to 77ba448 Compare June 14, 2022 04:56

Add support for resuming existing vdiff UUID

2d34490

What's left is adding the lastPK results to the SELECT used on the source and target tablets for retrieving the rows to compare. Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vdiff2_resume branch from e988504 to 2d34490 Compare June 14, 2022 06:31

mattlord added the Skip Upgrade Downgrade label Jun 14, 2022

mattlord changed the title ~~Support Resuming VDiffs~~ VDiff2: Support Resuming VDiffs Jun 14, 2022

mattlord added 2 commits June 14, 2022 13:16

Use v1 tablet_types default for v2

d026cd0

Signed-off-by: Matt Lord <mattalord@gmail.com>

Load the lastPK where it's needed, when building the SELECTs

79d081a

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vdiff2_resume branch 2 times, most recently from df3b50b to cb8a92e Compare June 15, 2022 18:55

Start passing lastpk to RowStreamers

30c1068

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vdiff2_resume branch from cb8a92e to 30c1068 Compare June 15, 2022 20:34

Update vdiff table's row estimate on resume

6f3b6ee

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord requested a review from rohit-nayak-ps June 15, 2022 23:44

mattlord force-pushed the vdiff2_resume branch from 115dad8 to 865c58b Compare June 16, 2022 00:13

mattlord force-pushed the vdiff2_resume branch from 865c58b to 3596741 Compare June 16, 2022 00:21

mattlord added 5 commits June 16, 2022 00:04

Align table_name and lastpk definition with copy_state table

484acbc

And add invariant check in updating table status Signed-off-by: Matt Lord <mattalord@gmail.com>

Merge remote-tracking branch 'origin/main' into vdiff2_resume

d26c6f4

Signed-off-by: Matt Lord <mattalord@gmail.com>

Update vdiff metadata on resume

5f41810

Signed-off-by: Matt Lord <mattalord@gmail.com>

Make getting lastPK for a table a utility function

5c6b292

Remove debugging code used during development. Signed-off-by: Matt Lord <mattalord@gmail.com>

Move getTableLastPK to workflowDiffer for connection re-use

4445f16

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the vdiff2_resume branch 3 times, most recently from 0b38f0e to 3c26cc1 Compare June 16, 2022 20:55

Manage states as typed consts and properly set vdiff.completed_timestamp

0653df9

Signed-off-by: Matt Lord <mattalord@gmail.com>

Minor comments

36e0056

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord requested a review from rohit-nayak-ps June 25, 2022 04:23

mattlord added 8 commits June 25, 2022 00:56

Minor tweaks after self review

a37f5e0

Signed-off-by: Matt Lord <mattalord@gmail.com>

RENAME COLUMN is new in 8.0

e92cf1c

Signed-off-by: Matt Lord <mattalord@gmail.com>

Avoid unnecessary intermediate slice (derrr)

42d48a2

Signed-off-by: Matt Lord <mattalord@gmail.com>

Remove lingering %w verb in log.*

2ac130c

Turns out that log does not support this verb. So we can stick to using it with fmt instead. Signed-off-by: Matt Lord <mattalord@gmail.com>

Reset rows_compared on resume

e22f27f

This provides accurate progress info during the resume run Signed-off-by: Matt Lord <mattalord@gmail.com>

Fix timout bug and other minor tweaks

c2724ab

Signed-off-by: Matt Lord <mattalord@gmail.com>

Fix VDiff summary completed bug AND add StartedAt summary output

a25f6f0

Signed-off-by: Matt Lord <mattalord@gmail.com>

Improvements around timestamp and global summary state handling

18ef02a

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord added Backport me! labels Jun 26, 2022

mattlord added 6 commits June 26, 2022 18:08

Add helpful comments and code grouping for summary building

36887cc

Signed-off-by: Matt Lord <mattalord@gmail.com>

Further improvements to State CompletedAt handling in Summary

c2f3083

Signed-off-by: Matt Lord <mattalord@gmail.com>

Align columns in table based summary output

62ecbe3

Signed-off-by: Matt Lord <mattalord@gmail.com>

Keep existing table based summary output order

58be2cf

And update release notes for new output Signed-off-by: Matt Lord <mattalord@gmail.com>

Add cli error handling tests

940d891

Since this was not working before this PR. :-) Signed-off-by: Matt Lord <mattalord@gmail.com>

Pedantry... halp

87a3d1c

Signed-off-by: Matt Lord <mattalord@gmail.com>

rohit-nayak-ps reviewed Jun 27, 2022

View reviewed changes

mattlord added 2 commits June 27, 2022 08:38

Address review comments

4baa244

Signed-off-by: Matt Lord <mattalord@gmail.com>

Merge branch 'main' into vdiff2_resume

f0c2873

mattlord requested a review from rohit-nayak-ps June 27, 2022 12:42

rohit-nayak-ps approved these changes Jun 27, 2022

View reviewed changes

mattlord mentioned this pull request Jun 27, 2022

Backport: VDiff2: Support Resuming VDiffs #10589

Merged

3 tasks

mattlord merged commit 67d1f38 into vitessio:main Jun 27, 2022

mattlord deleted the vdiff2_resume branch June 27, 2022 15:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VDiff2: Support Resuming VDiffs#10497

VDiff2: Support Resuming VDiffs#10497
mattlord merged 52 commits intovitessio:mainfrom
planetscale:vdiff2_resume

mattlord commented Jun 13, 2022 •

edited

Loading

Uh oh!

rohit-nayak-ps Jun 27, 2022

Uh oh!

mattlord Jun 27, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohit-nayak-ps Jun 27, 2022

Uh oh!

rohit-nayak-ps left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mattlord commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Uh oh!

rohit-nayak-ps Jun 27, 2022

Choose a reason for hiding this comment

Uh oh!

mattlord Jun 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohit-nayak-ps Jun 27, 2022

Choose a reason for hiding this comment

Uh oh!

rohit-nayak-ps left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mattlord commented Jun 13, 2022 •

edited

Loading

mattlord Jun 27, 2022 •

edited

Loading