Add a vtbackup batch command to take a backup and exit by dkhenry · Pull Request #4858 · vitessio/vitess

dkhenry · 2019-05-07T20:19:38Z

This adds a new vtbackup binary, which is a batch command that allows taking a backup without shutting down a running tablet. The vtbackup command will restore from the latest backup, silently replicate the latest changes without joining the Vitess cluster as a tablet, then take a new backup and exit.

Signed-off-by: Dan Kozlowski koz@planetscale.com

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

go/cmd/vttablet/vttablet.go

test/backup.py

Removing whitespace and renaming flag Signed-off-by: Dan Kozlowski <koz@planetscale.com>

test/backup.py

go/cmd/vttablet/vttablet.go

deepthi · 2019-05-08T17:21:35Z

Q: what is the motivation for this feature?

dkhenry · 2019-05-09T19:11:36Z

The motivation for this feature is to have the backup workflow of

Provision a new tablet
Restore from the Previous Backup, which will validate that the previous backup is functional
Take a new backup from the restored tablet
Cleanly exit

deepthi · 2019-05-09T20:52:24Z

The motivation for this feature is to have the backup workflow of

Provision a new tablet

Restore from the Previous Backup, which will validate that the previous backup is functional

Take a new backup from the restored tablet

Cleanly exit

This seems like a nice way to update your backups to be more current.
Is there any expectation around replica lag before taking a new backup?

dkhenry · 2019-05-09T20:58:30Z

There is. The busy loop that is there is to wait until the tablet catches up. It won't set SERVING until it is caught up, however on @enisoc 's suggestion I am swithing this to work with explicit BACKUP types so I am going to change the test slightly to be when SERVING would be set

… of tablet Signed-off-by: Dan Kozlowski <koz@planetscale.com>

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

sougou

I like how you did this. It's elegant and clean.

However, I have to agree with @enisoc. This brings in a fundamental change to what vttablet is: instead of being a server, it becomes a job. I think this will cause problems in the future.

This job has to be done from outside, ideally by an automation layer like a script or an operator.

This removes the backup_only from vttablet and creates a new binary only for backups. This binary does not register itsself with the topo server, but it does still require a valid tablet alias to facilitate replication. This can also be though of as Just Enough Vitess for Backup Signed-off-by: Dan Kozlowski <koz@planetscale.com>

dkhenry · 2019-05-11T01:22:27Z

@enisoc @deepthi - I totally redid this. There is now a vtbackup job that only takes backups. This will require some more cleanup and tests before it is safe to merge, but this should generally be an acceptable solution

enisoc · 2019-05-11T03:39:37Z

I haven't reviewed it in detail yet, but the new approach looks awesome! This seems like it'll have all the nice properties we wanted in the long-term solution. Let me know when you've got it ready for final review.

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

The lag value can lie. Also we're guaranteed to hit the snapshot position eventually as long as replication is making progress. If our goal is a lag value, we may never catch up if replication is slower than the rate of new transactions on the master. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Clean up and add policy enforcement

test/backup.py

deepthi

@enisoc @dkhenry can one of you update the PR description right at the top? It is no longer accurate.

Requiring that no tablets are running at all was too strict. It prevented using the new -wait_for_backup_interval flag to start up tablets concurrently while the initial backup is going. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

This makes it better match the state that you'd have if you used InitShardMaster instead. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Also adding in support for non initial_backup backup being taken with vtbackup Signed-off-by: Dan Kozlowski <koz@planetscale.com>

…k-backup-only Signed-off-by: Dan Kozlowski <koz@planetscale.com>

dkhenry · 2019-06-19T03:48:09Z

@deepthi The title and description look accurate.
@enisoc the tests have been updated. I had to made a few more changes to vtbackup and the tests. I added a new command line flag to allow this binary to take the first backup of an existing shard.

go/cmd/vtbackup/vtbackup.go

test/backup.py

test/backup_only.py

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

go/vt/mysqlctl/backup.go

Teardown option is no longer used Signed-off-by: Dan Kozlowski <koz@planetscale.com>

…-only Signed-off-by: Dan Kozlowski <koz@planetscale.com>

This block of code doesn't make any sense the way its written and I feel something has been lost in translation. I have modified it so at least I can explain it. If you have no backups you will return ErrNoBackup. If you fail to reset replication you will log that message If you fail to Populate the Metadata tables we will return that error Signed-off-by: Dan Kozlowski <koz@planetscale.com>

We need to create a backup with the new timestamp so we know that our backups are fresh. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

enisoc · 2019-06-28T06:56:29Z

@dkhenry FYI I added another commit to remove a check I had added earlier, which turned out to be problematic: 33b3035

By the way, is this ready for re-review?

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

dkhenry · 2019-06-28T17:57:07Z

@enisoc yes this is ready for re-review

go/vt/mysqlctl/backup.go

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

go/vt/mysqlctl/backup.go

test/backup.py

test/backup_only.py

test/backup.py

test/backup_only.py

deepthi

Looking good

go/vt/mysqlctl/backup.go

deepthi · 2019-06-29T03:20:34Z

test/backup_only.py

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

Adding a new option to vttablet to enable it to take a backup and exit

cae96e0

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

dkhenry requested a review from sougou as a code owner May 7, 2019 20:19

dkhenry requested review from deepthi and enisoc May 7, 2019 20:19

deepthi reviewed May 7, 2019

View reviewed changes

go/cmd/vttablet/vttablet.go Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

Addressing feeback from PR

68b83a9

Removing whitespace and renaming flag Signed-off-by: Dan Kozlowski <koz@planetscale.com>

enisoc requested changes May 8, 2019

View reviewed changes

test/backup.py Outdated Show resolved Hide resolved

go/cmd/vttablet/vttablet.go Outdated Show resolved Hide resolved

go/cmd/vttablet/vttablet.go Outdated Show resolved Hide resolved

Dan Kozlowski added 2 commits May 9, 2019 14:44

Moving backup job to its own file, and changing condition for restart…

fdaa16e

… of tablet Signed-off-by: Dan Kozlowski <koz@planetscale.com>

Add missing test file

3f74350

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

sougou reviewed May 10, 2019

View reviewed changes

enisoc changed the title ~~Adding a new option to vttablet to enable it to take a backup and exit~~ Add a vtbackup batch command to take a backup and exit May 29, 2019

enisoc added 6 commits June 4, 2019 16:18

vtbackup: Document and clean up.

af81ebc

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Incorporate mysqlctld behavior so that's not needed.

9a43d10

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Add to vitess/lite docker images.

e89e429

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Add -initial_backup flag for seeding an empty backup.

e10c624

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Add backup interval and pruning with retention policy.

22bccdd

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

deepthi mentioned this pull request Jun 5, 2019

vtbackup: Clean up and add policy enforcement planetscale/vitess#1

Merged

enisoc and others added 3 commits June 7, 2019 12:41

vtbackup: Make -initial_backup mode idempotent.

7e9e3ba

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

vtbackup: Add docker/k8s/vtbackup image.

62fd75c

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Merge pull request #1 from planetscale/enisoc-vtbackup

3151386

vtbackup: Clean up and add policy enforcement

dkhenry requested a review from derekperkins as a code owner June 10, 2019 21:44

deepthi reviewed Jun 10, 2019

View reviewed changes

test/backup.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

deepthi suggested changes Jun 10, 2019

View reviewed changes

enisoc and others added 4 commits June 14, 2019 15:32

vtbackup: Also create the main database in initial backup.

71eaa6f

This makes it better match the state that you'd have if you used InitShardMaster instead. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Modifying tests to support refactor of vtbackup.

f756576

Also adding in support for non initial_backup backup being taken with vtbackup Signed-off-by: Dan Kozlowski <koz@planetscale.com>

Merge branch 'dk-backup-only' of github.com:planetscale/vitess into d…

16ab3ff

…k-backup-only Signed-off-by: Dan Kozlowski <koz@planetscale.com>

enisoc reviewed Jun 19, 2019

View reviewed changes

Merge branch 'master' of github.com:vitessio/vitess into dk-backup-only

8200553

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

deepthi suggested changes Jun 20, 2019

View reviewed changes

go/vt/mysqlctl/backup.go Outdated Show resolved Hide resolved

Removing teardown option from backup file

4e1e9bc

Teardown option is no longer used Signed-off-by: Dan Kozlowski <koz@planetscale.com>

dkhenry requested review from deepthi and enisoc June 28, 2019 02:20

Dan Kozlowski and others added 3 commits June 27, 2019 19:22

Merge branch 'master' of github.com:planetscale/vitess into dk-backup…

44e1dff

…-only Signed-off-by: Dan Kozlowski <koz@planetscale.com>

Still take a new backup even if nothing changed.

33b3035

We need to create a backup with the new timestamp so we know that our backups are fresh. Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Allow backup to proceed even if nothing changed on master.

0b58b99

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

deepthi suggested changes Jun 28, 2019

View reviewed changes

go/vt/mysqlctl/backup.go Outdated Show resolved Hide resolved

Updating backup logic

3286680

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

enisoc approved these changes Jun 29, 2019

View reviewed changes

go/vt/mysqlctl/backup.go Show resolved Hide resolved

deepthi suggested changes Jun 29, 2019

View reviewed changes

test/backup.py Outdated Show resolved Hide resolved

test/backup_only.py Outdated Show resolved Hide resolved

test/backup.py Outdated Show resolved Hide resolved

test/backup_only.py Show resolved Hide resolved

deepthi approved these changes Jun 29, 2019

View reviewed changes

go/vt/mysqlctl/backup.go Show resolved Hide resolved

test/backup_only.py Outdated

Copy link
Copy Markdown

Collaborator

deepthi Jun 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Implementing feedback from PR

9264759

Signed-off-by: Dan Kozlowski <koz@planetscale.com>

dkhenry force-pushed the dk-backup-only branch from 4d8ffe0 to 9264759 Compare June 29, 2019 03:43

dkhenry merged commit 22c64df into vitessio:master Jun 29, 2019

Conversation

dkhenry commented May 7, 2019 • edited by enisoc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepthi commented May 8, 2019

Uh oh!

dkhenry commented May 9, 2019

Uh oh!

deepthi commented May 9, 2019

Uh oh!

dkhenry commented May 9, 2019

Uh oh!

sougou left a comment

Choose a reason for hiding this comment

Uh oh!

dkhenry commented May 11, 2019

Uh oh!

enisoc commented May 11, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepthi left a comment

Choose a reason for hiding this comment

Uh oh!

dkhenry commented Jun 19, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

enisoc commented Jun 28, 2019

Uh oh!

dkhenry commented Jun 28, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepthi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deepthi Jun 29, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dkhenry commented May 7, 2019 •

edited by enisoc

Loading