…te jobs.

This shown up as a problem under parallel roslyn.

We hit frequently the case of multiple threads compiling the same method in parallel
and wasting all but one.

Another issue that happens, but infrequently, is having more threads JITing than there
are cores in the machine. Such thrashing doesn't help.

Test setup. 4/8 macbook pro compiling corlib.

Baseline:
real	0m7.665s
user	0m20.078s
sys	0m2.653s
Methods JITted using mono JIT       : 22422
Total time spent JITting (sec)      : 18.5365

With this patch:
real	0m6.149s
user	0m18.504s
sys	0m1.487s
Methods JITted using mono JIT       : 16619
Total time spent JITting (sec)      : 4.9420

New counters
JIT compile waited others           : 7681
JIT compile 1+ jobs                 : 1
JIT compile overload wait           : 67
JIT compile spurious wakeups        : 14469

This results in a 20% wall clock reduction but only a 8% reduction on user.

We JIT 26% less methods, with very few duplications. Showing this drastically improves the situation.

JIT compilation time metrics are bogus due to .cctors and other sources of interference. So take it with a grain of salt.

Future work:

Based on the new counters, it's clear that the current wakeup design is suboptimal and we could further improve it by cutting on spurious wakeups.

The first iteration of this code suffered from spurious wakeups due to using a single cond-var for all compilation.

This introduce a per-job cond var and make threads wait on that. Spurious wakeups are quite problematic in heavily threaded
code such as Roslyn. On average, every wait would get 2 extra wakeups for no good reason.

With this change, on my laptop compilation times are the following:

With spurious wakeups:
real	0m6.134s
user	0m23.588s
sys	0m2.331s

Without spurious wakeups:
real	0m5.658s
user	0m22.851s
sys	0m1.694s

This represents a 8% wallclock speedup and 3% user time speedup.

As part of this change, the active threads accounting is gone as the number of waits due to cpu overload was close to zero.