Skip to content

Managed DROP TABLE#6719

Merged
shlomi-noach merged 25 commits intovitessio:masterfrom
planetscale:managed-drop-table
Oct 6, 2020
Merged

Managed DROP TABLE#6719
shlomi-noach merged 25 commits intovitessio:masterfrom
planetscale:managed-drop-table

Conversation

@shlomi-noach
Copy link
Copy Markdown
Contributor

@shlomi-noach shlomi-noach commented Sep 14, 2020 •
edited
Loading

Tracking issue: #6689

Implementing a TableGC (garbage collector) component as part of TabletServer, which periodically checks for tables in various table-drop states, transitions the tables between those states, purges data from the tables, or drops them altogether.

At this time this PR is not concerned with user facing DROP statements.

@shlomi-noach shlomi-noach marked this pull request as ready for review September 17, 2020 07:01
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Sep 17, 2020

This PR is now ready for review. Still TODO:

  • endtoend tests
  • accompanying documentation

@shlomi-noach shlomi-noach changed the title WIP: Managed DROP TABLE Managed DROP TABLE Sep 17, 2020
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Sep 17, 2020

Please note this PR requires, and is based on, #6668 . It is best if we first merge #6668. Reviewing this PR will become easier.

@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Sep 17, 2020

This PR implements most of the DROP TABLE logic in #6689 : it runs the table lifecycle behind the scenes, down to dropping the table. The only part it does not implement is the user facing DROP IN '24h' TABLE my_table syntax. This will be taken care of by a followup PR.

This PR is enough to run table garbage collection, if only the user/app know how to RENAME their table into the correct magic form. That's easy to do if you use the mechanisms go/vt/schema. And this is exactly what #6547 (or one of its followups) will do at the end of migration, for the artifact tables.

This was referenced Sep 17, 2020
Merged
Closed
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Sep 21, 2020

endtoend tests now added

@shlomi-noach shlomi-noach mentioned this pull request Sep 29, 2020
Closed
@shlomi-noach
WIP: table dropper (aka garbage collector)
3a840a8 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
tablegc supportive functions
fcca82c 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
added Evac state, added tests, table name analysis
2b34a8b 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
rename hint->state. Parsing lifecycle comma delimites state string
e706362 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
fix state_manager_test
4ec225d 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
initial state machine implementation in TableGC (renamed from TableDr…
ac541ba 
…opper)

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
added unit test
3dd6aa5 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactor to better state machine logic, now tested
5cbbb1e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
purge logic in place
0fdeda1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
actual purge of data (still need to apply SQL_LOG_bin=0
5756dfb 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
clearing and setting SQL_LOG_BIN
9173b00 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
feature complete. SQL_LOG_BIN=0 when purging rows
c56d91c 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
more logging, DROP implicitly included in lifecycle
dae8c2e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
adding endtoend test file
c50b442 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
adding shard 27, running tablegc on shard 27
905f8cd 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
debug message cleanup
ddb1783 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
removed redundant Sleep
d103395 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
moved suspendable_ticker around
c6e7ffc 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
using refactored SuspendableTicker
8ae0fb1 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
fixed Open()/Close() behavior
b75629b 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactored RandomHash into new textutil package
a8b772b 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Oct 1, 2020

This PR is in good shape the way I see it. It works through a table's lifecycle to eventually drop it; it only works on tables with "magic" names like _vt_HOLD_6ace8bcef73211ea87e9f875a4d24e90_20200915120410.

It does not offer a DROP TABLE syntax -- that's for a different PR.

So, this PR only introduces a new behavior for very special tables, which will be immediately put to use by #6547 .

@shlomi-noach shlomi-noach requested a review from deepthi October 1, 2020 10:21
@shlomi-noach
removed util.go
411809f 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
refactored list parsing function
9c177de 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach mentioned this pull request Oct 5, 2020
Merged
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Oct 5, 2020

I'm ready to have this PR reviewed and merged.

deepthi
deepthi approved these changes Oct 5, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me.
We can merge after the conflicts are resolved and tests pass.

@shlomi-noach
merged master, resolved conflicts
07443aa 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
resolved error, state manager tests
d41a656 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Oct 6, 2020

conflicts resolved, tests pass. Per review, I'm now merging (!)

@shlomi-noach shlomi-noach merged commit 7ad166f into vitessio:master Oct 6, 2020
@shlomi-noach shlomi-noach deleted the managed-drop-table branch October 6, 2020 06:54
@deepthi deepthi mentioned this pull request Oct 6, 2020
Merged
@askdba askdba added this to the v8.0 milestone Oct 7, 2020
@setassociative setassociative mentioned this pull request Mar 5, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@derekperkins derekperkins Awaiting requested review from derekperkins

@dkhenry dkhenry Awaiting requested review from dkhenry

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal

@sougou sougou Awaiting requested review from sougou

@systay systay Awaiting requested review from systay

1 more reviewer

@deepthi deepthi deepthi approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v8.0

Development

Successfully merging this pull request may close these issues.

3 participants

@shlomi-noach @deepthi @askdba