Today, the tablet lag throttler is configured y vttablet command line flags, and notably:
--enable-lag-throttler--throttle_threshold
(other flags also apply).
Essentially, either the lag is enabled upon tablet startup, and then remains enabled, or it is disabled and can never run. Likewise, the threshold is only set upon startup.
We want to be able to make changes to the throttler on a running tablet. Specifically, we're interested in commands such as these:
alter vitess_throttler disable - disable the throttler on all shards, if runningalter vitess_throttler enable - enable throttler on all shards, if not already runningalter vitess_throttler threshold '2s' - change threshold on all shardsalter vitess_throttler reload - reload and apply settings from command line flagsshow vitess_throttler status - get cuent state: enabled/disabled, threshold
etc.
These commands must survive: a tablet restart, PRS and ERS, or else this leads to surprising behavior, user experience-wise.
To support the setup survival, we will introduce a _vt.throttler_config table with such columns as enabled, threshold, etc.
The behavior will be such:
- Upon startup, if the table is empty, the tablet will populate it by command line flags
- Whether the table was already populated or the tablet populated it as per above, the table reads the config data from the table, and applies the state to the throttler
- Obviously, the table changes get replicated to all shard's replicas.
- Any time the throttler is
Open()ed (by tm_state), it re-reads table data and applies as needed
- This implicitly means any PRS/ERS will auto-read last known configuration
- A
alter vitess_throttler reload command overwrites table data with command line flag configuration, and proceeds to apply the state as needed.
show vitess_throttler status reads and returns config from backend table
The reads/writes to backend table are minimal and only take place upon:
Open() (implicitly including startup)- User initiated
alter vitess_throttler ... commands
And so there is no operational overhead to using a backend table.
The current throttler code, derived from freno, assumes the throttler either starts when the table starts and then runs forever, or never runs. We need to make logic changes to accommodate enable/disable. e.g. destroying timers, flushing caches, protecting some variables with mutexes.
There is WIP branch called alter-vitess-throttler, not yet a PR, which begins to implement everything discussed in this issue.
cc @vitessio/ps-vitess for review
Today, the tablet lag throttler is configured y
vttabletcommand line flags, and notably:--enable-lag-throttler--throttle_threshold(other flags also apply).
Essentially, either the lag is enabled upon tablet startup, and then remains enabled, or it is disabled and can never run. Likewise, the threshold is only set upon startup.
We want to be able to make changes to the throttler on a running tablet. Specifically, we're interested in commands such as these:
alter vitess_throttler disable- disable the throttler on all shards, if runningalter vitess_throttler enable- enable throttler on all shards, if not already runningalter vitess_throttler threshold '2s'- change threshold on all shardsalter vitess_throttler reload- reload and apply settings from command line flagsshow vitess_throttler status- get cuent state: enabled/disabled, thresholdetc.
These commands must survive: a tablet restart, PRS and ERS, or else this leads to surprising behavior, user experience-wise.
To support the setup survival, we will introduce a
_vt.throttler_configtable with such columns asenabled,threshold, etc.The behavior will be such:
Open()ed (bytm_state), it re-reads table data and applies as neededalter vitess_throttler reloadcommand overwrites table data with command line flag configuration, and proceeds to apply the state as needed.show vitess_throttler statusreads and returns config from backend tableThe reads/writes to backend table are minimal and only take place upon:
Open()(implicitly including startup)alter vitess_throttler ...commandsAnd so there is no operational overhead to using a backend table.
The current throttler code, derived from freno, assumes the throttler either starts when the table starts and then runs forever, or never runs. We need to make logic changes to accommodate enable/disable. e.g. destroying timers, flushing caches, protecting some variables with mutexes.
There is WIP branch called
alter-vitess-throttler, not yet a PR, which begins to implement everything discussed in this issue.cc @vitessio/ps-vitess for review