Skip to content

Conversation

@jnyrup
Copy link
Member

@jnyrup jnyrup commented Mar 29, 2018

I tried to benchmark BeEquivalentTo as that seems to the most heavy assertion due to heavy use of reflection.

The most visible bottleneck was recreating the delegate for TraceBlock in LooselyMatchAgainst in each iteration, instead of caching it.
It used 24.15% of the time.

There are also a lot of places, where we convert an IEnumerable to a new Array/List, but it seems we could just cast the object if it actually is an ICollection or IList,
So I created helper functions ConvertOrCastToCollection and ConvertOrCastToList to help reduce the amount of unnecessary copying.

Here's the code I used to test two lists of 20.000 identical 6-nested objects.

class Complex
{
    public int A { get; set; }
    public Complex B { get; set; }
}

private static Complex CreateComplex(int i)
{
    if (i == 0)
    {
        return new Complex();
    }

    return new Complex
    {
        A = i,
        B = CreateComplex(i - 1)
    };
}

private const int N = 20_000;
private const int Depth = 6;

private readonly List<Complex> list = Enumerable.Range(0, N).Select(i => CreateComplex(Depth)).ToList();
private readonly List<Complex> list2 = Enumerable.Range(0, N).Select(i => CreateComplex(Depth)).ToList();

[TestMethod]
public void lol()
{
    list.Should().BeEquivalentTo(list2);
}

@Meir017
Copy link
Member

Meir017 commented Mar 29, 2018

Could we perhaps add some benchmark tests to the repo similar to the ones you ran?

@dennisdoomen
Copy link
Member

What did you use? DotTrace?

@jnyrup
Copy link
Member Author

jnyrup commented Mar 29, 2018

I used the builtin profiler in Visual Studio 2017.

@jnyrup
Copy link
Member Author

jnyrup commented Mar 30, 2018

@Meir017 Do you have an idea how to structure this?

My thoughts:

  • Create a new project only for benchmarks
  • Add benchmarks using BenchmarkDotNet
  • Run benchmarks before and after a commit to verify improvement

@Meir017
Copy link
Member

Meir017 commented Mar 30, 2018

yup 👍 I was thinking of maybe using existing tests and executing them multiple times

public class CollectionEquivalencySpecsBenchmark
{
    public CollectionEquivalencySpecs Subject { get; }
    public CollectionEquivalencySpecsBenchmark()
    {
        Subject = new CollectionEquivalencySpecs();
    }

    [Benchmark]
    public void When_a_collection_does_not_match_it_should_include_items_in_message()
        => Subject.When_a_collection_does_not_match_it_should_include_items_in_message();
}

and of course we could generate this code :)

@dennisdoomen
Copy link
Member

Now let's hope AppVeyor uses the same kind of machine every time.

@jnyrup
Copy link
Member Author

jnyrup commented Mar 30, 2018

I wouldn't count on AppVeyor using the same kind of machine every time.
https://bheisler.github.io/post/benchmarking-in-the-cloud/

The best approach I can currently think of, it setting up a benchmark baseline before doing improvements and running it locally before and after to verify improvements.

Ideally it would require a dedicated benchmark machine, but I think that's out of scope.

@jnyrup jnyrup force-pushed the Optimizations branch 3 times, most recently from 95a6c96 to de9cdbd Compare March 31, 2018 07:53
@jnyrup
Copy link
Member Author

jnyrup commented Mar 31, 2018

I'll be adding more benchmarks before merging this, but here are the results of before and after the Cache Delegate commit.

Before:

Total time: 00:19:41 (1181.17 sec)

// * Summary *

BenchmarkDotNet=v0.10.13, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.309)
Intel Core i7-6700HQ CPU 2.60GHz (Skylake), 1 CPU, 8 logical cores and 4 physical cores
Frequency=2531250 Hz, Resolution=395.0617 ns, Timer=TSC
  [Host]    : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0
  RyuJitX86 : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0

Job=RyuJitX86  Jit=RyuJit  Platform=X86

         Method |     N |           Mean |        Error |       StdDev |       Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|-------------:|-------------:|------------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       986.2 us |     2.728 us |     2.130 us |     72.2656 |        - |       - |    226.57 KB |
 BeEquivalentTo |   100 |     7,366.1 us |    65.619 us |    61.380 us |    703.1250 |        - |       - |   2181.47 KB |
 BeEquivalentTo |  1000 |    78,753.8 us |   548.075 us |   512.670 us |   9312.5000 |        - |       - |  28619.89 KB |
 BeEquivalentTo |  5000 |   609,422.6 us | 5,042.880 us | 4,211.033 us |  97437.5000 |        - |       - | 299459.54 KB |
 BeEquivalentTo | 10000 | 1,740,293.6 us | 2,390.986 us | 1,866.724 us | 321812.5000 | 437.5000 | 62.5000 | 989739.01 KB |
Total time: 00:14:01 (841.46 sec)

// * Summary *

BenchmarkDotNet=v0.10.13, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.309)
Intel Core i7-6700HQ CPU 2.60GHz (Skylake), 1 CPU, 8 logical cores and 4 physical cores
Frequency=2531250 Hz, Resolution=395.0617 ns, Timer=TSC
  [Host]    : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0
  RyuJitX86 : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0

Job=RyuJitX86  Jit=RyuJit  Platform=X86

         Method |     N |           Mean |       Error |      StdDev |      Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|------------:|------------:|-----------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       990.3 us |    14.69 us |    13.74 us |    72.2656 |        - |       - |    225.16 KB |
 BeEquivalentTo |   100 |     7,380.7 us |    81.85 us |    76.57 us |   679.6875 |        - |       - |   2096.35 KB |
 BeEquivalentTo |  1000 |    74,842.8 us |   391.43 us |   366.14 us |  6687.5000 |        - |       - |  20729.87 KB |
 BeEquivalentTo |  5000 |   491,089.2 us |   714.07 us |   667.94 us | 33750.0000 |        - |       - | 103924.52 KB |
 BeEquivalentTo | 10000 | 1,272,079.7 us | 2,473.93 us | 1,931.48 us | 67250.0000 | 437.5000 | 62.5000 | 207984.15 KB |

@jnyrup
Copy link
Member Author

jnyrup commented Apr 16, 2018

TLDR;

  • N = 10

    • Speed improved by 1.6x
    • Memory reduced by 2.3x
  • N = 10,000

    • Speed improved by 4.8x
    • Memory reduced by 6.4x

baseline:

	 Method |     N |           Mean |        Error |       StdDev |       Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|-------------:|-------------:|------------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       986.2 us |     2.728 us |     2.130 us |     72.2656 |        - |       - |    226.57 KB |
 BeEquivalentTo |   100 |     7,366.1 us |    65.619 us |    61.380 us |    703.1250 |        - |       - |   2181.47 KB |
 BeEquivalentTo |  1000 |    78,753.8 us |   548.075 us |   512.670 us |   9312.5000 |        - |       - |  28619.89 KB |
 BeEquivalentTo |  5000 |   609,422.6 us | 5,042.880 us | 4,211.033 us |  97437.5000 |        - |       - | 299459.54 KB |
 BeEquivalentTo | 10000 | 1,740,293.6 us | 2,390.986 us | 1,866.724 us | 321812.5000 | 437.5000 | 62.5000 | 989739.01 KB |

Cache Deletegate - 4140064

         Method |     N |           Mean |       Error |      StdDev |      Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|------------:|------------:|-----------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       990.3 us |    14.69 us |    13.74 us |    72.2656 |        - |       - |    225.16 KB |
 BeEquivalentTo |   100 |     7,380.7 us |    81.85 us |    76.57 us |   679.6875 |        - |       - |   2096.35 KB |
 BeEquivalentTo |  1000 |    74,842.8 us |   391.43 us |   366.14 us |  6687.5000 |        - |       - |  20729.87 KB |
 BeEquivalentTo |  5000 |   491,089.2 us |   714.07 us |   667.94 us | 33750.0000 |        - |       - | 103924.52 KB |
 BeEquivalentTo | 10000 | 1,272,079.7 us | 2,473.93 us | 1,931.48 us | 67250.0000 | 437.5000 | 62.5000 | 207984.15 KB |

Combine Linq expressions - 6c0adea

         Method |     N |           Mean |         Error |        StdDev |      Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|--------------:|--------------:|-----------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       946.8 us |      8.662 us |      7.233 us |    66.4063 |        - |       - |    205.79 KB |
 BeEquivalentTo |   100 |     6,787.5 us |     63.642 us |     59.531 us |   617.1875 |        - |       - |   1903.03 KB |
 BeEquivalentTo |  1000 |    69,735.6 us |    537.478 us |    448.818 us |  6062.5000 |        - |       - |  18808.24 KB |
 BeEquivalentTo |  5000 |   462,402.3 us |  3,124.722 us |  2,609.285 us | 30625.0000 |        - |       - |  94314.18 KB |
 BeEquivalentTo | 10000 | 1,220,280.6 us | 14,163.785 us | 11,058.151 us | 61000.0000 | 437.5000 | 62.5000 | 188799.49 KB | 

Cast enumerable to list/collection - e01e0d7

         Method |     N |           Mean |        Error |       StdDev |      Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|-------------:|-------------:|-----------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       933.5 us |     8.726 us |     7.736 us |    66.4063 |        - |       - |    205.79 KB |
 BeEquivalentTo |   100 |     6,691.3 us |    52.743 us |    49.336 us |   617.1875 |        - |       - |   1903.03 KB |
 BeEquivalentTo |  1000 |    69,248.4 us |   484.353 us |   453.064 us |  6062.5000 |        - |       - |  18808.24 KB |
 BeEquivalentTo |  5000 |   460,448.2 us | 2,383.609 us | 2,113.007 us | 30625.0000 |        - |       - |  94314.18 KB |
 BeEquivalentTo | 10000 | 1,227,924.0 us | 2,987.471 us | 2,648.315 us | 61000.0000 | 437.5000 | 62.5000 | 188799.49 KB |

Optimize HasValueSemantics - 9abdff3

         Method |     N |           Mean |       Error |       StdDev |      Gen 0 |    Gen 1 |   Gen 2 |    Allocated |
--------------- |------ |---------------:|------------:|-------------:|-----------:|---------:|--------:|-------------:|
 BeEquivalentTo |    10 |       788.3 us |    10.20 us |     9.542 us |    62.5000 |        - |       - |    192.36 KB |
 BeEquivalentTo |   100 |     5,504.2 us |    64.29 us |    56.992 us |   578.1250 |        - |       - |   1778.15 KB |
 BeEquivalentTo |  1000 |    58,873.0 us | 1,174.48 us | 1,567.895 us |  5687.5000 |        - |       - |  17613.13 KB |
 BeEquivalentTo |  5000 |   401,444.3 us |   941.78 us |   834.861 us | 28687.5000 |        - |       - |  88393.29 KB |
 BeEquivalentTo | 10000 | 1,091,310.8 us | 2,671.20 us | 2,230.577 us | 57062.5000 | 437.5000 | 62.5000 | 176874.67 KB |

Use a list of unmatched indexes - cd719c3

         Method |     N |         Mean |         Error |        StdDev |       Median |      Gen 0 |    Allocated |
--------------- |------ |-------------:|--------------:|--------------:|-------------:|-----------:|-------------:|
 BeEquivalentTo |    10 |     784.9 us |      7.183 us |      6.367 us |     784.8 us |    61.5234 |    192.05 KB |
 BeEquivalentTo |   100 |   5,391.9 us |     55.781 us |     46.580 us |   5,382.1 us |   570.3125 |   1773.46 KB |
 BeEquivalentTo |  1000 |  50,857.4 us |    468.106 us |    437.866 us |  50,806.0 us |  5687.5000 |  17559.24 KB |
 BeEquivalentTo |  5000 | 260,260.3 us |    576.385 us |    539.151 us | 260,438.8 us | 28687.5000 |  88156.71 KB |
 BeEquivalentTo | 10000 | 534,215.8 us | 10,643.436 us | 24,667.782 us | 513,440.4 us | 57375.0000 | 176390.39 KB |

Combine() - 3d3cca7

         Method |     N |         Mean |        Error |       StdDev |      Gen 0 |    Allocated |
--------------- |------ |-------------:|-------------:|-------------:|-----------:|-------------:|
 BeEquivalentTo |    10 |     762.1 us |     8.088 us |     6.754 us |    58.5938 |    182.36 KB |
 BeEquivalentTo |   100 |   5,037.4 us |    61.900 us |    57.901 us |   539.0625 |   1677.21 KB |
 BeEquivalentTo |  1000 |  48,334.2 us |   471.410 us |   440.957 us |  5375.0000 |  16604.24 KB |
 BeEquivalentTo |  5000 | 238,672.2 us |   610.323 us |   570.896 us | 27125.0000 |  83331.18 KB |
 BeEquivalentTo | 10000 | 543,754.3 us | 3,745.026 us | 3,319.867 us | 54250.0000 | 166742.26 KB |

Cache HasValueSemantics - 95738ed

         Method |     N |         Mean |        Error |       StdDev |      Gen 0 |    Allocated |
--------------- |------ |-------------:|-------------:|-------------:|-----------:|-------------:|
 BeEquivalentTo |    10 |     724.2 us |     7.792 us |     6.908 us |    56.6406 |    176.35 KB |
 BeEquivalentTo |   100 |   4,685.8 us |    61.590 us |    57.611 us |   523.4375 |   1616.34 KB |
 BeEquivalentTo |  1000 |  43,512.5 us |   468.911 us |   438.619 us |  5187.5000 |  15991.73 KB |
 BeEquivalentTo |  5000 | 220,138.7 us |   626.710 us |   555.562 us | 26125.0000 |  80284.60 KB |
 BeEquivalentTo | 10000 | 476,748.2 us | 5,532.648 us | 4,319.527 us | 52250.0000 | 160693.09 KB |

Cache OverridesEquals - 12bd297

         Method |     N |         Mean |        Error |       StdDev |      Gen 0 |    Allocated |
--------------- |------ |-------------:|-------------:|-------------:|-----------:|-------------:|
 BeEquivalentTo |    10 |     633.1 us |     7.848 us |     7.341 us |    54.6875 |    170.86 KB |
 BeEquivalentTo |   100 |   3,948.4 us |    58.894 us |    52.208 us |   500.0000 |   1558.02 KB |
 BeEquivalentTo |  1000 |  35,757.8 us |   405.432 us |   379.242 us |  5000.0000 |   15402.2 KB |
 BeEquivalentTo |  5000 | 180,096.6 us |   555.060 us |   492.046 us | 25125.0000 |   77342.1 KB |
 BeEquivalentTo | 10000 | 366,110.4 us | 1,146.773 us | 1,016.585 us | 50375.0000 | 154756.81 KB |

@jnyrup jnyrup merged commit b4d5880 into fluentassertions:master Apr 18, 2018
@jnyrup jnyrup deleted the Optimizations branch April 20, 2018 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants