Online stats sizes and specifying exact slab class sizes#169
Closed
dormando wants to merge 3 commits into
Closed
Conversation
"stats sizes" is one of the lack cache-hanging commands. With millions of items it can hang for many seconds. This commit changes the command to be dynamic. A histogram is tracked as items are linked and unlinked from the cache. The tracking is enabled or disabled at runtime via "stats sizes_enable" and "stats sizes_disable". This presently "works" but isn't accurate. Giving it some time to think over before switching to requiring that CAS be enabled. Otherwise the values could underflow if items are removed that existed before the sizes tracker is enabled. This attempts to work around it by using it->time, which gets updated on fetch, and is thus inaccurate.
"-o slab_sizes=100-200-300-400-500" will create 5 slab classes of those specified sizes, with the final class being item_max_size. Using the new online stats sizes command, it's possible to determine if the typical factoral slab class growth rate doesn't align well with how items are stored. This is dangerous unless you really know what you're doing. If your items have an exact or very predictable size this makes a lot of sense. If they do not, the defaults are safer.
Now relies on CAS feature for runtime enable/disable tracking. Still usable if enabled at starttime with CAS disabled. Also adds start option `-o track_sizes`, and a stat for `stats settings`. Finally, adds documentation and cleans up status outputs. Could use some automated tests but not make or break for release.
Member
Author
|
This is now complete so far as I'm concerned. I won't merge it for a day or two while I work on other things. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This two-parter PR is handy for folks wanting to tune their slab class sizes.
The
stats sizescommand was generally off limits as it used to hang the cache. More recently it just hangs that particular worker thread and hammers the locks.With this update a passive system can be enabled or disabled live which will track a histogram of item sizes as they're added or removed from the cache. This allows temporarily enabling it, or simply enable and leave it running if you don't mind the overhead. The command then dumps just the pre-created histogram on demand.
stats sizes_enableandstats sizes_disableenable and disable the feature.If your
-fand-noptions can be tuned to give less memory overhead, that's great. However, what if you have a cluster where items only exist at exact sizes, or the factorial is never quite close enough?You can now specify the slab classes manually via
-o slab_sizes=100-200-300-400-500. Which will create 5 slab classes around those sizes, plus a sixth at the max item size.TODO:
Optional: