When pystats are collected inside of the benchmarking infrastructure, pystats is turned off during pyperf's warmup and calibration steps. This means by the time the actual benchmarking happens and the stats are turned on, the code may already be specialized and traced and moved to tier 2. So we miss out on trace creation attempt stats, for example, this is from json_dumps -- we are executing some traces, but not creating any, which doesn't make sense until you realize they were created during "warmup".

This effect has existed for sometime and is undercounting our specialization attempts as well, but it's more obviously missing for trace creation.

I think the easy fix is to just always turn stats collection on, even for the warmup phases, which will increase the overall stats counts, but it probably more realistic. A more involved change to pyperf would be to disable the warmup and calibration when doing a stats collection.

@markshannon: Does this make sense? How much do we care about this problem?