Ready for review

This PR implements a "forward" alter table via VReplication.

General design described in #6926 (comment), which this PR implements.

Notable bullets about the changes in this PR:

• The main logic of this PR is in go/vt/vttablet/onlineddl/vrepl.go and go/vt/vttablet/onlineddl/executor.go

• go/vt/vttablet/onlineddl/executor.go can now invoke, cancel and terminate VReplication based migrations. Some of this management is symmetric to how we invoke, cancel and terminate gh-ost or pt-osc, but there are also significant differences:

  • while gh-ost and pt-osc can only work within the context of this tablet (invoked as an OS Exec), VReplication migrations can execute by one tablet and continued on another, even in face of failover.
  • gh-ost and pt-osc have self-logic to run the migration. VReplication needs to be told what exactly it should run. The tablet knows about a alter table statement, and needs to create a matching rule/filter query for VReplication.
  • gh-ost and pt-osc have self-logic to cut-over. Vreplication does not. Executor identifies the occasion for cut-over and implements the cut-over logic, which entails stopping writes to original tables, waiting for pos, stopping VReplication, replacing tables.

• Imported a bunch of code from https://github.com/github/gh-ost. These are found under go/vt/vttablet/onlineddl/vrepl. The kind of functionality in those files:

  • Parsing an ALTER TABLE statement
  • Structures to define columns
  • Dealing with datatypes
    We don't strictly need all the above as-is. We can replace parsing with vitess's parsing. We don't need to track datatypes because vreplication does. But I copied&pasted those files because they are mature and stable and get the work done.
    I intend to refactor them later on and remove redundant/unrelated code, or replace with existing Vitess functionality. But I'd like to do so on a separate PR so as to focus on the main overall functionality here.

• Split onlineddl/endtoend tests into endtoend_ghost and endtoend_vrepl. They are mostly similar but with some changes (like how do you throttle a migration). The tests are OK-ish but need to be more elaborate. I again wish to follow up in a future PR as I am yet to design a satisfactory endtoend test for schema migrations and failure scenarios.

• Due to some earlier attempt to use wrangler inside tabletserver, I created a new interface called vexec.Executor, and refactored ddlExecutor to be vexec.Executor instead of onlineddl.Executor. I ended up not using wrangler inside tabletserver but I liked the refactor and kept it.

initial PR comment when this was Work In Progress:

Description

Implementing Online DDL via VReplication.
There's a bunch of code to wrap the VReplication mechanism. I import a lot of logic from gh-ost (analyzing tables, analyzing primary key, sanity checks). Some of this should be replaced with vitess's sqlparser, but for now it's faster to go with gh-ost imported code as I know it is stable.

This is work in progrss. the idea is that we run VReplication in a materialize-like fashion, where both source and target are the shard's PRIMARY.

Vreplication is kicked by the online DDL executor, and not by vtctl/wrangler. this is very different from the normal VReplication flow, and reasoning is that we want the tablets to validate and to schedule the migration at its own good time. So we're using the existing Online DDL Executor logic to start/stop migrations. Vreplication is different in that it can survive restarts or failovers. This is something the executor will take into account.

VReplication only works through PRIMARY KEYs. In that aspect, gh-ost is more flexible as it can iterate any UNIQUE KEY. Another thing for me to check is what are valid PRIMARY KEY changes: is it OK to add a column to a PRIMARY KEY? Remove a column?

Another difference to normal VReplication flow is how we do the cut-over. We will automate the cut-over, and will then rename the tables. No change to routing rules. Later, we'd need to see if we can also reverse VReplication for renamed tables. I'm not sure how simple/hard that would be.

anyway, this is really work in progress and no guarantees. Will comment as progress is made.

Related Issue(s)

• tracking: Online DDL: tracking issue #6926

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build/CI
• VTAdmin