Skip to content

enable full percentile spectrum analysis on latency#100

Merged
yossigo merged 4 commits intoredis:masterfrom
filipecosta90:hdr.hist
Aug 12, 2020
Merged

enable full percentile spectrum analysis on latency#100
yossigo merged 4 commits intoredis:masterfrom
filipecosta90:hdr.hist

Conversation

@filipecosta90
Copy link
Collaborator

@filipecosta90 filipecosta90 commented Nov 10, 2019 •
edited
Loading

Full percentile spectrum analysis

Fixes: #68
This PR enable full percentile spectrum analysis on query latency so that we can fully understand the performance and stability characteristics of the system we are measuring.

Right now memtier_benchmark is displaying single values targets like the average.
Considering that latency does not obey a normal distribution we should enable ourselves to do full percentile spectrum analysis. Just looking at a number might be misleading.

A useful visualization for pointing out the folly of chasing a mean measurement is illustrated below - they have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed.

image

Source: https://en.wikipedia.org/wiki/Anscombe%27s_quartet

Enabling full percentile latencies on memtier_benchmark:

The previous features can be applied with the changes added on this PR using the following added options:

      --hdr-file-prefix=FILE     Prefix of HDR Latency Histogram output files, if not set, will not save latency histogram files
      --print-percentiles          Specify which percentiles info to print on the results table (by default prints percentiles: 50,99,99.9)

yossigo
yossigo suggested changes Nov 17, 2019
View reviewed changes
Copy link
Contributor

@yossigo yossigo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@filipecosta90 Thanks for this PR, I've went through most of it (maybe not enough in detail in some parts though) and added a few comments.
Nitpicking: there are some extra white space and code style issues in this PR, it would be a good idea to re-review it and clean it up.

yaacovhazan-Redislabs
yaacovhazan-Redislabs suggested changes Nov 21, 2019
View reviewed changes
yaacovhazan-Redislabs
yaacovhazan-Redislabs suggested changes Dec 4, 2019
View reviewed changes
@filipecosta90 filipecosta90 changed the title enable full percentile spectrum analysis on latency. enable maintaining fixed rps during query benchmarks enable full percentile spectrum analysis on latency Jan 27, 2020
@filipecosta90
Copy link
Collaborator Author

filipecosta90 commented Aug 9, 2020

Hi there @yossigo and @yaacovhazan-Redislabs I've pruned the PR and I believe it's ready for a new review.
I've updated the hdrhistogram to the latest official one ( v0.11.0 ), added a test case for the added features, and extended the PR to store per second latency/throughput metrics at the output json ( The stored percentiles comes from the --print-percentiles property ). Here is a snippet:

,"ALL STATS":{
		"Runtime":{
			"Start time": 1597016193374
			,"Finish time": 1597016207579
			,"Total duration": 14205
			,"Time unit": "MILLISECONDS"
			}
		,"Sets":{
			"Count": 182000
			,"Ops/sec": 12812.51
			,"Hits/sec": 0.00
			,"Misses/sec": 0.00
			,"Latency": 1.419
			,"Average Latency": 1.419
			,"Min Latency": 0.032
			,"Max Latency": 14.415
			,"KB/sec": 986.79
			,"Time-Serie":{
				"0":{
					"Count": 14225
					,"Average Latency": 1.29
					,"Min Latency": 0.54
					,"Max Latency": 6.43
					,"p50.00": 1.17
					,"p99.00": 3.73
					,"p99.90": 4.96
					}
				,"1":{
					"Count": 14739
					,"Average Latency": 1.23
					,"Min Latency": 0.10
					,"Max Latency": 7.78
					,"p50.00": 1.20
					,"p99.00": 2.62
					,"p99.90": 5.86
					}
(...)

WDYT?

@yossigo yossigo merged commit 145c02b into redis:master Aug 12, 2020
@yossigo
Copy link
Contributor

yossigo commented Aug 12, 2020

@filipecosta90 Thanks a lot for this effort! It's a great improvement and definitely worth the effort and time invested in it.

@filipecosta90 filipecosta90 mentioned this pull request Aug 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@yossigo yossigo yossigo approved these changes

@yaacovhazan-Redislabs yaacovhazan-Redislabs yaacovhazan-Redislabs approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Provide more latency data in results

3 participants

@filipecosta90 @yossigo @yaacovhazan-Redislabs