Skip to content

Resurrect legacy benchmarks#69

Merged
curiousleo merged 2 commits intoinstances-and-warningsfrom
legacy-benchmark
Apr 3, 2020
Merged

Resurrect legacy benchmarks#69
curiousleo merged 2 commits intoinstances-and-warningsfrom
legacy-benchmark

Conversation

@curiousleo
Copy link
Copy Markdown
Collaborator

@curiousleo curiousleo commented Apr 3, 2020 

$ stack bench random:legacy-bench
[...]
Running 1 benchmarks...
Benchmark legacy-bench: RUNNING...
                    
How many random numbers can we generate in a second on one thread?
  Cost of rdtsc (ffi call):    19
  Approx getCPUTime calls per second: 1,063,978
  Approx clock frequency:  1,564,593,291
  First, timing System.Random.next:
    153,771,354 randoms generated [constant zero gen]         ~ 10.17 cycles/int
     68,698,046 randoms generated [System.Random stdGen/next] ~ 22.77 cycles/int
                    
  Second, timing System.Random.random at different types:
     48,466,626 randoms generated [System.Random Ints]        ~ 32.28 cycles/int
     58,428,650 randoms generated [System.Random Word16]      ~ 26.78 cycles/int
     59,836,036 randoms generated [System.Random Floats]      ~ 26.15 cycles/int
     63,707,504 randoms generated [System.Random CFloats]     ~ 24.56 cycles/int
     75,634,028 randoms generated [System.Random Doubles]     ~ 20.69 cycles/int
     69,857,837 randoms generated [System.Random CDoubles]    ~ 22.40 cycles/int
      1,141,199 randoms generated [System.Random Integers]    ~ 1,371 cycles/int
      1,211,491 randoms generated [System.Random Bools]       ~ 1,291 cycles/int
      1,026,950 randoms generated [System.Random Chars]       ~ 1,524 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
      1,362,845 randoms generated [System.Random Ints]        ~ 1,148 cycles/int
     15,983,047 randoms generated [System.Random Word16s]     ~ 97.89 cycles/int
     43,625,018 randoms generated [System.Random Floats]      ~ 35.86 cycles/int
     59,511,662 randoms generated [System.Random CFloats]     ~ 26.29 cycles/int
     47,344,655 randoms generated [System.Random Doubles]     ~ 33.05 cycles/int
     52,693,665 randoms generated [System.Random CDoubles]    ~ 29.69 cycles/int
      1,800,875 randoms generated [System.Random Integers]    ~ 869 cycles/int
        982,026 randoms generated [System.Random Bools]       ~ 1,593 cycles/int
      1,028,566 randoms generated [System.Random Chars]       ~ 1,521 cycles/int
         19,559 randoms generated [System.Random BIG Integers] ~ 79,994 cycles/int
                    
Now 8 threads, reporting mean randoms-per-second-per-thread:
  First, timing System.Random.next:
     37,851,319 randoms generated [constant zero gen]         ~ 41.34 cycles/int
     17,083,419 randoms generated [System.Random stdGen/next] ~ 91.59 cycles/int
                    
  Second, timing System.Random.random at different types:
     17,716,627 randoms generated [System.Random Ints]        ~ 88.31 cycles/int
     14,350,964 randoms generated [System.Random Word16]      ~ 109 cycles/int
     15,813,176 randoms generated [System.Random Floats]      ~ 98.94 cycles/int
     16,336,176 randoms generated [System.Random CFloats]     ~ 95.77 cycles/int
     20,212,208 randoms generated [System.Random Doubles]     ~ 77.41 cycles/int
     18,437,271 randoms generated [System.Random CDoubles]    ~ 84.86 cycles/int
        283,124 randoms generated [System.Random Integers]    ~ 5,526 cycles/int
        378,537 randoms generated [System.Random Bools]       ~ 4,133 cycles/int
        270,908 randoms generated [System.Random Chars]       ~ 5,775 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
        428,136 randoms generated [System.Random Ints]        ~ 3,654 cycles/int
      5,902,902 randoms generated [System.Random Word16s]     ~ 265 cycles/int
     14,419,648 randoms generated [System.Random Floats]      ~ 109 cycles/int
     18,736,332 randoms generated [System.Random CFloats]     ~ 83.51 cycles/int
     17,833,027 randoms generated [System.Random Doubles]     ~ 87.74 cycles/int
     19,346,497 randoms generated [System.Random CDoubles]    ~ 80.87 cycles/int
        615,296 randoms generated [System.Random Integers]    ~ 2,543 cycles/int
        332,982 randoms generated [System.Random Bools]       ~ 4,699 cycles/int
        378,384 randoms generated [System.Random Chars]       ~ 4,135 cycles/int
          6,590 randoms generated [System.Random BIG Integers] ~ 237,415 cycles/int
Finished.           
Benchmark legacy-bench: FINISH

@lehins
Copy link
Copy Markdown
Collaborator

lehins commented Apr 3, 2020 •
edited
Loading

Guys, I think we did an outstanding job of improving performance of random!!!!!
I just ran the same benchmarks on master:

How many random numbers can we generate in a second on one thread?
  Cost of rdtsc (ffi call):    58
  Approx getCPUTime calls per second: 1,183,626
  Approx clock frequency:  2,091,452,897
  First, timing System.Random.next:
    150,197,153 randoms generated [constant zero gen]         ~ 13.92 cycles/int
     14,712,243 randoms generated [System.Random stdGen/next] ~ 142 cycles/int
                    
  Second, timing System.Random.random at different types:
      1,392,852 randoms generated [System.Random Ints]        ~ 1,502 cycles/int
      4,864,392 randoms generated [System.Random Word16]      ~ 430 cycles/int
      1,922,820 randoms generated [System.Random Floats]      ~ 1,088 cycles/int
      1,917,856 randoms generated [System.Random CFloats]     ~ 1,091 cycles/int
      1,036,642 randoms generated [System.Random Doubles]     ~ 2,018 cycles/int
      2,145,254 randoms generated [System.Random CDoubles]    ~ 975 cycles/int
      1,416,475 randoms generated [System.Random Integers]    ~ 1,477 cycles/int
      3,290,266 randoms generated [System.Random Bools]       ~ 636 cycles/int
      3,364,515 randoms generated [System.Random Chars]       ~ 622 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
      3,203,552 randoms generated [System.Random Ints]        ~ 653 cycles/int
      3,063,546 randoms generated [System.Random Word16s]     ~ 683 cycles/int
        983,564 randoms generated [System.Random Floats]      ~ 2,126 cycles/int
      1,939,157 randoms generated [System.Random CFloats]     ~ 1,079 cycles/int
        642,928 randoms generated [System.Random Doubles]     ~ 3,253 cycles/int
      1,957,399 randoms generated [System.Random CDoubles]    ~ 1,068 cycles/int
      4,285,473 randoms generated [System.Random Integers]    ~ 488 cycles/int
      3,093,533 randoms generated [System.Random Bools]       ~ 676 cycles/int
      3,140,335 randoms generated [System.Random Chars]       ~ 666 cycles/int
         16,033 randoms generated [System.Random BIG Integers] ~ 130,447 cycles/int
                    
Now 8 threads, reporting mean randoms-per-second-per-thread:
  First, timing System.Random.next:
     84,023,331 randoms generated [constant zero gen]         ~ 24.89 cycles/int
      7,616,805 randoms generated [System.Random stdGen/next] ~ 275 cycles/int
                    
  Second, timing System.Random.random at different types:
        760,066 randoms generated [System.Random Ints]        ~ 2,752 cycles/int
      2,645,208 randoms generated [System.Random Word16]      ~ 791 cycles/int
      1,044,121 randoms generated [System.Random Floats]      ~ 2,003 cycles/int
      1,178,424 randoms generated [System.Random CFloats]     ~ 1,775 cycles/int
        624,218 randoms generated [System.Random Doubles]     ~ 3,351 cycles/int
      1,333,682 randoms generated [System.Random CDoubles]    ~ 1,568 cycles/int
        813,524 randoms generated [System.Random Integers]    ~ 2,571 cycles/int
      1,850,364 randoms generated [System.Random Bools]       ~ 1,130 cycles/int
      1,909,855 randoms generated [System.Random Chars]       ~ 1,095 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
      1,810,916 randoms generated [System.Random Ints]        ~ 1,155 cycles/int
      1,710,549 randoms generated [System.Random Word16s]     ~ 1,223 cycles/int
        543,232 randoms generated [System.Random Floats]      ~ 3,850 cycles/int
      1,168,755 randoms generated [System.Random CFloats]     ~ 1,789 cycles/int
        379,622 randoms generated [System.Random Doubles]     ~ 5,509 cycles/int
      1,120,228 randoms generated [System.Random CDoubles]    ~ 1,867 cycles/int
      2,354,608 randoms generated [System.Random Integers]    ~ 888 cycles/int
      1,766,973 randoms generated [System.Random Bools]       ~ 1,184 cycles/int
      1,750,330 randoms generated [System.Random Chars]       ~ 1,195 cycles/int
          6,305 randoms generated [System.Random BIG Integers] ~ 331,700 cycles/int
Finished.           
Benchmark legacy-bench: FINISH
```

lehins
lehins approved these changes Apr 3, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@lehins lehins left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI is failing probably cause rdtsc needs to be add to other stack.yaml files (Looked at the wrong commit)
But otherwise looks great

@lehins
Copy link
Copy Markdown
Collaborator

lehins commented Apr 3, 2020

In case you want to run those benchmarks on master on your machine, here is the branch you can use: https://github.com/idontgetoutmuch/random/tree/legacy-bench-master

@curiousleo
Copy link
Copy Markdown
Collaborator Author

curiousleo commented Apr 3, 2020

The CI is failing probably cause rdtsc needs to be add to other stack.yaml files

Fixed - CI has passed now: https://travis-ci.org/github/idontgetoutmuch/random/builds/670532338

But otherwise looks great

:D

@curiousleo
Copy link
Copy Markdown
Collaborator Author

curiousleo commented Apr 3, 2020 •
edited
Loading

In case you want to run those benchmarks on master on your machine, here is the branch you can use: https://github.com/idontgetoutmuch/random/tree/legacy-bench-master

Excellent. I'll post the results here so we can compare numbers obtained from the same machine.

Edit: here's what I get on master (aka random v1.1):

$ stack bench random:legacy-bench
[...]
random> benchmarks
Running 1 benchmarks...
Benchmark legacy-bench: RUNNING...
                    
How many random numbers can we generate in a second on one thread?
  Cost of rdtsc (ffi call):    20
  Approx getCPUTime calls per second: 941,827
  Approx clock frequency:  1,426,853,886
  First, timing System.Random.next:
    168,084,590 randoms generated [constant zero gen]         ~ 8.49 cycles/int
     13,956,750 randoms generated [System.Random stdGen/next] ~ 102 cycles/int
                    
  Second, timing System.Random.random at different types:
      1,593,080 randoms generated [System.Random Ints]        ~ 896 cycles/int
      4,874,622 randoms generated [System.Random Word16]      ~ 293 cycles/int
      2,468,696 randoms generated [System.Random Floats]      ~ 578 cycles/int
      2,368,020 randoms generated [System.Random CFloats]     ~ 603 cycles/int
      1,287,323 randoms generated [System.Random Doubles]     ~ 1,108 cycles/int
      2,677,939 randoms generated [System.Random CDoubles]    ~ 533 cycles/int
      1,676,903 randoms generated [System.Random Integers]    ~ 851 cycles/int
      3,562,326 randoms generated [System.Random Bools]       ~ 401 cycles/int
      3,537,625 randoms generated [System.Random Chars]       ~ 403 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
      3,692,188 randoms generated [System.Random Ints]        ~ 386 cycles/int
      3,505,322 randoms generated [System.Random Word16s]     ~ 407 cycles/int
      1,203,388 randoms generated [System.Random Floats]      ~ 1,186 cycles/int
      2,315,522 randoms generated [System.Random CFloats]     ~ 616 cycles/int
        805,961 randoms generated [System.Random Doubles]     ~ 1,770 cycles/int
      2,383,191 randoms generated [System.Random CDoubles]    ~ 599 cycles/int
      4,504,970 randoms generated [System.Random Integers]    ~ 317 cycles/int
      3,333,011 randoms generated [System.Random Bools]       ~ 428 cycles/int
      3,329,210 randoms generated [System.Random Chars]       ~ 429 cycles/int
         13,103 randoms generated [System.Random BIG Integers] ~ 108,895 cycles/int
                    
Now 8 threads, reporting mean randoms-per-second-per-thread:
  First, timing System.Random.next:
     77,572,774 randoms generated [constant zero gen]         ~ 18.39 cycles/int
      7,504,086 randoms generated [System.Random stdGen/next] ~ 190 cycles/int
                    
  Second, timing System.Random.random at different types:
        750,957 randoms generated [System.Random Ints]        ~ 1,900 cycles/int
      2,272,808 randoms generated [System.Random Word16]      ~ 628 cycles/int
        987,246 randoms generated [System.Random Floats]      ~ 1,445 cycles/int
        820,021 randoms generated [System.Random CFloats]     ~ 1,740 cycles/int
        457,140 randoms generated [System.Random Doubles]     ~ 3,121 cycles/int
        897,434 randoms generated [System.Random CDoubles]    ~ 1,590 cycles/int
        525,219 randoms generated [System.Random Integers]    ~ 2,717 cycles/int
      1,192,638 randoms generated [System.Random Bools]       ~ 1,196 cycles/int
      1,305,256 randoms generated [System.Random Chars]       ~ 1,093 cycles/int
                    
  Next timing range-restricted System.Random.randomR:
      1,215,496 randoms generated [System.Random Ints]        ~ 1,174 cycles/int
      1,227,125 randoms generated [System.Random Word16s]     ~ 1,163 cycles/int
        421,746 randoms generated [System.Random Floats]      ~ 3,383 cycles/int
        783,029 randoms generated [System.Random CFloats]     ~ 1,822 cycles/int
        295,027 randoms generated [System.Random Doubles]     ~ 4,836 cycles/int
        621,758 randoms generated [System.Random CDoubles]    ~ 2,295 cycles/int
      1,000,299 randoms generated [System.Random Integers]    ~ 1,426 cycles/int
        750,803 randoms generated [System.Random Bools]       ~ 1,900 cycles/int
      1,016,428 randoms generated [System.Random Chars]       ~ 1,404 cycles/int
          5,898 randoms generated [System.Random BIG Integers] ~ 241,922 cycles/int
Finished.           
Benchmark legacy-bench: FINISH

@curiousleo curiousleo merged commit 9574756 into instances-and-warnings Apr 3, 2020
@curiousleo curiousleo deleted the legacy-benchmark branch April 6, 2020 07:22
@curiousleo
Copy link
Copy Markdown
Collaborator Author

curiousleo commented Apr 6, 2020

Comparison: 1.1 on the left, instances-and-warnings on the right:

How many random numbers can we generate in a second on one thread?
  Cost of rdtsc (ffi call):    20                                                    19
  Approx getCPUTime calls per second: 941,827                                        1,063,978
  Approx clock frequency:  1,426,853,886                                             1,564,593,291
  First, timing System.Random.next:
    168,084,590 randoms generated [constant zero gen]         ~ 8.49 cycles/int      ~ 10.17 cycles/int
     13,956,750 randoms generated [System.Random stdGen/next] ~ 102 cycles/int       ~ 22.77 cycles/int
                    	                    
  Second, timing System.Random.random at different types:
      1,593,080 randoms generated [System.Random Ints]        ~ 896 cycles/int       ~ 32.28 cycles/int
      4,874,622 randoms generated [System.Random Word16]      ~ 293 cycles/int       ~ 26.78 cycles/int
      2,468,696 randoms generated [System.Random Floats]      ~ 578 cycles/int       ~ 26.15 cycles/int
      2,368,020 randoms generated [System.Random CFloats]     ~ 603 cycles/int       ~ 24.56 cycles/int
      1,287,323 randoms generated [System.Random Doubles]     ~ 1,108 cycles/int     ~ 20.69 cycles/int
      2,677,939 randoms generated [System.Random CDoubles]    ~ 533 cycles/int       ~ 22.40 cycles/int
      1,676,903 randoms generated [System.Random Integers]    ~ 851 cycles/int       ~ 1,371 cycles/int
      3,562,326 randoms generated [System.Random Bools]       ~ 401 cycles/int       ~ 1,291 cycles/int
      3,537,625 randoms generated [System.Random Chars]       ~ 403 cycles/int       ~ 1,524 cycles/int
                    	                    
  Next timing range-restricted System.Random.randomR:
      3,692,188 randoms generated [System.Random Ints]        ~ 386 cycles/int       ~ 1,148 cycles/int
      3,505,322 randoms generated [System.Random Word16s]     ~ 407 cycles/int       ~ 97.89 cycles/int
      1,203,388 randoms generated [System.Random Floats]      ~ 1,186 cycles/int     ~ 35.86 cycles/int
      2,315,522 randoms generated [System.Random CFloats]     ~ 616 cycles/int       ~ 26.29 cycles/int
        805,961 randoms generated [System.Random Doubles]     ~ 1,770 cycles/i       ~ 33.05 cycles/int
      2,383,191 randoms generated [System.Random CDoubles]    ~ 599 cycles/int       ~ 29.69 cycles/int
      4,504,970 randoms generated [System.Random Integers]    ~ 317 cycles/int       ~ 869 cycles/int
      3,333,011 randoms generated [System.Random Bools]       ~ 428 cycles/int       ~ 1,593 cycles/int
      3,329,210 randoms generated [System.Random Chars]       ~ 429 cycles/int       ~ 1,521 cycles/int
         13,103 randoms generated [System.Random BIG Integers] ~ 108,895 cycles/int  ~ 79,994 cycles/int
                    	                    
Now 8 threads, reporting mean randoms-per-second-per-thread:
  First, timing System.Random.next:
     77,572,774 randoms generated [constant zero gen]         ~ 18.39 cycles/int     ~ 41.34 cycles/int
      7,504,086 randoms generated [System.Random stdGen/next] ~ 190 cycles/int       ~ 91.59 cycles/int
                    	                    
  Second, timing System.Random.random at different types:
        750,957 randoms generated [System.Random Ints]        ~ 1,900 cycles/int     ~ 88.31 cycles/int
      2,272,808 randoms generated [System.Random Word16]      ~ 628 cycles/int       ~ 109 cycles/int
        987,246 randoms generated [System.Random Floats]      ~ 1,445 cycles/int     ~ 98.94 cycles/int
        820,021 randoms generated [System.Random CFloats]     ~ 1,740 cycles/int     ~ 95.77 cycles/int
        457,140 randoms generated [System.Random Doubles]     ~ 3,121 cycles/int     ~ 77.41 cycles/int
        897,434 randoms generated [System.Random CDoubles]    ~ 1,590 cycles/int     ~ 84.86 cycles/int
        525,219 randoms generated [System.Random Integers]    ~ 2,717 cycles/int     ~ 5,526 cycles/int
      1,192,638 randoms generated [System.Random Bools]       ~ 1,196 cycles/int     ~ 4,133 cycles/int
      1,305,256 randoms generated [System.Random Chars]       ~ 1,093 cycles/int     ~ 5,775 cycles/int
                    	                    
  Next timing range-restricted System.Random.randomR:
      1,215,496 randoms generated [System.Random Ints]        ~ 1,174 cycles/int     ~ 3,654 cycles/int
      1,227,125 randoms generated [System.Random Word16s]     ~ 1,163 cycles/int     ~ 265 cycles/int
        421,746 randoms generated [System.Random Floats]      ~ 3,383 cycles/int     ~ 109 cycles/int
        783,029 randoms generated [System.Random CFloats]     ~ 1,822 cycles/int     ~ 83.51 cycles/int
        295,027 randoms generated [System.Random Doubles]     ~ 4,836 cycles/int     ~ 87.74 cycles/int
        621,758 randoms generated [System.Random CDoubles]    ~ 2,295 cycles/int     ~ 80.87 cycles/int
      1,000,299 randoms generated [System.Random Integers]    ~ 1,426 cycles/int     ~ 2,543 cycles/int
        750,803 randoms generated [System.Random Bools]       ~ 1,900 cycles/int     ~ 4,699 cycles/int
      1,016,428 randoms generated [System.Random Chars]       ~ 1,404 cycles/int     ~ 4,135 cycles/int
          5,898 randoms generated [System.Random BIG Integers] ~ 241,922 cycles/int  ~ 237,415 cycles/int

curiousleo added a commit that referenced this pull request May 19, 2020
@curiousleo
Merge pull request #69 from idontgetoutmuch/legacy-benchmark
bbbdbab 
Resurrect legacy benchmarks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lehins lehins lehins approved these changes

@idontgetoutmuch idontgetoutmuch Awaiting requested review from idontgetoutmuch

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@curiousleo @lehins