Skip to content

Throttler: Store Config in Global Keyspace Topo Record#12520

Merged
deepthi merged 22 commits intovitessio:mainfrom
planetscale:throttler_topo_fixes
Mar 8, 2023
Merged

Throttler: Store Config in Global Keyspace Topo Record#12520
deepthi merged 22 commits intovitessio:mainfrom
planetscale:throttler_topo_fixes

Conversation

@mattlord
Copy link
Copy Markdown
Member

@mattlord mattlord commented Feb 28, 2023 •
edited
Loading

Description

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required

@mattlord
Store throttler config in global Keyspace record
89727e9 
Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord mattlord added Type: Bug Component: Online DDL Online DDL (vitess/native/gh-ost/pt-osc) labels Feb 28, 2023
@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels Feb 28, 2023
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot bot commented Feb 28, 2023

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
  • If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@mattlord
Fix unit test after ThrottlerConfig in Keyspace
ce34f61 
Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord mattlord force-pushed the throttler_topo_fixes branch from bf67af0 to ce34f61 Compare February 28, 2023 19:53
@mattlord mattlord mentioned this pull request Feb 28, 2023
Closed
@mattlord
Apply changes lost in cherry picks
b4c347f 
Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord mattlord force-pushed the throttler_topo_fixes branch from baf15a0 to b4c347f Compare March 1, 2023 00:47
@mattlord
Always wait for a status before the final check.
7c1e958 
We should not ever make an assumption about how quickly
an operation will take.

Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord
Propagate PartialResult and other errs back to client
a0d612e 
Signed-off-by: Matt Lord <mattalord@gmail.com>
shlomi-noach
shlomi-noach reviewed Mar 7, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@shlomi-noach shlomi-noach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this! I think the main question (also found inline): if we update SrvKeyspace, and that in turn updates Keyspace, does the change propagate back down to all other SrvKeyspaces? Should it? Let's discuss.

@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
287921e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach added the Skip CI Skip CI actions from running label Mar 8, 2023
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
06bec1a 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
5d6185b 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
07925c6 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
9cc9176 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
69e53fb 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
26c4390 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
e4dce19 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
3ff4fd5 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
e58832e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
6d129c9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
9ef565b 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
c45fea5 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
e570fab 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Update go/test/endtoend/onlineddl/vtgate_util.go
3e4304c 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
fix grammar
e9b0d3a 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach removed the Skip CI Skip CI actions from running label Mar 8, 2023
@shlomi-noach shlomi-noach marked this pull request as ready for review March 8, 2023 05:58
shlomi-noach
shlomi-noach approved these changes Mar 8, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@shlomi-noach shlomi-noach left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've applied a few minor suggestions (remove excessive waits in testing; use context for timeout) -- let's merge this! (assuming tests are good)

@deepthi deepthi removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels Mar 8, 2023
@deepthi
Copy link
Copy Markdown
Collaborator

deepthi commented Mar 8, 2023

Both the test failures are flaky and there are people looking into them.

@deepthi
Copy link
Copy Markdown
Collaborator

deepthi commented Mar 8, 2023

I applied the Backport: label to take advantage of the bot. If we decide not to put this into release-16.0 we can close the bot-created PR.

@deepthi deepthi merged commit 0817f32 into vitessio:main Mar 8, 2023
@deepthi deepthi deleted the throttler_topo_fixes branch March 8, 2023 19:14
@vitess-bot vitess-bot bot mentioned this pull request Mar 8, 2023
Merged
deepthi pushed a commit that referenced this pull request Mar 9, 2023
@vitess-bot @mattlord @shlomi-noach
[release-16.0] Throttler: Store Config in Global Keyspace Topo Record (…
6e1f08a 
…#12520) (#12576)

* Store throttler config in global Keyspace record

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Fix unit test after ThrottlerConfig in Keyspace

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Apply changes lost in cherry picks

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Always wait for a status before the final check.

We should not ever make an assumption about how quickly
an operation will take.

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Propagate PartialResult and other errs back to client

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vtctlutil.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* Update go/test/endtoend/onlineddl/vtgate_util.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* fix grammar

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

---------

Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@mattlord mattlord mentioned this pull request Apr 20, 2023
Closed
@shlomi-noach shlomi-noach mentioned this pull request Apr 20, 2023
Closed
@shlomi-noach shlomi-noach mentioned this pull request May 23, 2023
Merged
4 tasks
This was referenced May 31, 2023
Closed
Merged
@hmaurer hmaurer mentioned this pull request Mar 21, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shlomi-noach shlomi-noach shlomi-noach approved these changes

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

@systay systay Awaiting requested review from systay systay is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps is a code owner

@ajm188 ajm188 Awaiting requested review from ajm188

@notfelineit notfelineit Awaiting requested review from notfelineit

@rsajwani rsajwani Awaiting requested review from rsajwani

+1 more reviewer

@deepthi deepthi deepthi approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: Online DDL Online DDL (vitess/native/gh-ost/pt-osc) Type: Bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug Report: Throttler Config Via Topo Is Broken

3 participants

@mattlord @deepthi @shlomi-noach