MangoHud icon indicating copy to clipboard operation
MangoHud copied to clipboard

Add mangoplot script

Open AdelKS opened this issue 3 years ago • 0 comments

Hello guys, as I started benchmarking with log_interval=0 to have accurate percentiles (MangoHud otherwise simply saves to disk 1 frame every log_interval), the produced files are too big to be handled by the website. A local way to plot and compare between benchmarks was lacking, so here it is.

This PR introduces a python script that enables comparing visually between benchmarks, and more accurately than what the website displays. It depends on matplotlib and needs a graphical back-end, I randomly chose PyQt5, we can decide on something else.

Here's an example output of the script (on a set of benchmarks I did)

benchmarks

It shows the distribution of the frame rates (1 / (frame time)), the 50%, the 1% and 0.1% percentiles. The n% percentiles are not exactly the n% lows: the n% lows involves averaging all the frame rates below the n% percentile.

Visualizing the distribution of the frame times enables seeing:

  • More nuances between benchmarks to properly assess what one prefers in one's own use-case
  • That a benchmark is not reproducible yet, if e.g. a rebench doesn't visually have the same distribution

we can even switch to this view on the website, maybe, visualizing the frames is pretty, but pretty un-useful, except maybe to confirm that two benchmarks are related by seeing that they have the same trends.

The script probably needs some polishing and maybe some extra features like:

  • Give which percentiles to display
  • Accept a list of benchmark files instead of only a folder.
  • Set an fps_min and an fps_max to crop the view (although it can be done with user input on the matplotlib window)

Feedback welcome !

AdelKS avatar Aug 28 '22 21:08 AdelKS