Skip to content

More array benchmarks#451

Merged
aspiwack merged 9 commits intomasterfrom
benchmarks/more-arrays
May 4, 2023
Merged

More array benchmarks#451
aspiwack merged 9 commits intomasterfrom
benchmarks/more-arrays

Conversation

@aspiwack
Copy link
Copy Markdown
Member

@aspiwack aspiwack commented May 3, 2023

  • Benchmark raw allocation
  • Benchmark writes
  • Benchmark against the Seq type

Still a draft because I need to format all this, and maybe I need to revisit a commit message: it would seem that the overhead that my refactoring introduced essentially compensates the difference between Vector and our linear arrays. Which proves that the extra Ur and (,) are probably where the issue is coming from (probably mostly Ur since, as we can see in #328 , the (,) are unboxed. So having the ability to unbox Ur will really be critical for the extra bit of performance).

That being said, there is something to be said for comparing the same scenarios. There is also something to be said for comparing idiomatic scenarios. In neither case is the comparison completely accurate anyway.

Here are my figures after this PR (note how good we are with writes, the figures are not too shabby honestly):

  arrays
    alloc
      Data.Array.Mutable.Linear:                                                 OK (0.41s)
        329  ns ±  29 ns
      Data.Vector:                                                               OK (0.45s)
        451  ns ±  23 ns
      Data.Sequence:                                                             OK (0.29s)
        3.88 μs ± 377 ns
    toList
      Data.Array.Mutable.Linear:                                                 OK (0.36s)
        5.70 μs ± 384 ns
      Data.Vector:                                                               OK (0.35s)
        5.82 μs ± 347 ns
      Data.Sequence:                                                             OK (0.58s)
        3.02 μs ±  95 ns
    map
      Data.Array.Mutable.Linear:                                                 OK (0.32s)
        2.24 μs ± 220 ns
      Data.Vector:                                                               OK (0.33s)
        2.62 μs ± 166 ns
      Data.Sequence:                                                             OK (1.14s)
        7.20 μs ± 436 ns
    reads
      Data.Array.Mutable.Linear:                                                 OK (0.44s)
        9.11 μs ± 355 ns
      Data.Vector:                                                               OK (0.32s)
        11.3 μs ± 755 ns
      Data.Sequence:                                                             OK (0.44s)
        73.8 μs ± 2.7 μs
    successive writes (very unfair to vector)
      Data.Array.Mutable.Linear:                                                 OK (0.37s)
        13.6 μs ± 759 ns
      Data.Vector:                                                               OK (0.57s)
        1.93 ms ± 184 μs
      Data.Sequence:                                                             OK (0.21s)
        116  μs ±  11 μs

aspiwack added 7 commits May 3, 2023 15:42
@aspiwack
Help for the future peerers-at-Core
bbe73e5
@aspiwack
A more reasonable size for arrays
09e6262 
We probably have a bit of overhead due to unsafe casts. So it's more
fair to have a reasonably sized array. The micro-benchmarking aspect
is taken care of by our benchmarking framework, so the precision
should still be as good. And it allows for slower tests.
@aspiwack
Add array benchmark for alloc and sets
f5f53f6
@aspiwack
Array benchmarks: use same implementation for both data structures
4c1a859 
I simply introduced a type class to abstract over the differences. I
almost didn't do it because I was afraid that the extra `Ur` and `(,)`
would pollute the result and bias the result in our favours. Maybe it
does. The reads are a bit less different in this style. I guess it was
to be expected because reads are so fast. But the benchmark still says
that our implementation is slow. So I say: good enough.
@aspiwack
Array benchmarks: clean-up refactoring
7fa9f08
@aspiwack
Array benchmark: refactor to avoid a little noise in test decls
0346704
@aspiwack
Array benchmarks: compare against Sequence
50a72f5
@aspiwack aspiwack force-pushed the benchmarks/more-arrays branch from 25f62bb to 6c2e181 Compare May 4, 2023 07:57
@aspiwack aspiwack marked this pull request as ready for review May 4, 2023 07:57
@aspiwack aspiwack requested a review from tbagrel1 May 4, 2023 07:58
@aspiwack
Copy link
Copy Markdown
Member Author

aspiwack commented May 4, 2023

Should be good now.

tbagrel1
tbagrel1 approved these changes May 4, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@tbagrel1 tbagrel1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I can convince myself that this code benchmarks each collection type without introducing extra noise that would impact performance.

That being said, at a syntactic/organisational level, the code is quite readable.

aspiwack added 2 commits May 4, 2023 16:52
@aspiwack
Array benchmarks: clean-up
90dd6f0
@aspiwack
Array benchmarks: allow more optimisations in scenarios
09a25b2 
Using a single version of the code has created less tight
benchmarks. This lets more specialisation happen. Though I can't say
it's perfect either.
@aspiwack aspiwack force-pushed the benchmarks/more-arrays branch from 6c2e181 to 09a25b2 Compare May 4, 2023 14:53
@aspiwack
Copy link
Copy Markdown
Member Author

aspiwack commented May 4, 2023

I'm not sure I can convince myself that this code benchmarks each collection type without introducing extra noise that would impact performance.

It does introduce noise. Now the question of whether it models actual real-life performance better or worse is much harder. I don't really know.

I've added a commit that helps with inlining and specialisation to remove some of that noise.

@aspiwack aspiwack enabled auto-merge May 4, 2023 14:56
@aspiwack aspiwack merged commit 66a9bd9 into master May 4, 2023
@aspiwack aspiwack deleted the benchmarks/more-arrays branch May 4, 2023 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tbagrel1 tbagrel1 tbagrel1 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aspiwack @tbagrel1