The github syntax highlighting is going nuts on this - is this a variation of JSON that has comments?

I guess? I just used the pandas version as a template.

Do we need to repeat it many times, or can we configure asv to do that?

I'm curious how big a role caching is going to play here and how misleading that will be.

asv does run each of the benchmarks multiple times, I think targeting a total run time of 0.1 seconds by default (configurable). I haven't mastered the semantics it uses. In some circumstances asv spawns subprocesses to avoid caching influencing the results.

Originally I didn't include these loops at all, but in the profiling it only ran once and I wanted a bigger sample.

@jbrockmendel Interesting. I wonder if it is better to just trust asv to run the task multiple times as it sees fit or if it's better to do some averaging on top of their averaging.

It is my suspicion that we'll want the bulk of our tests to be single-run tests and let asv be smart about how to run them, but since caching is a reality we'll want to deliberately induce it with some sort of loop at some times.

@pablogsal Do you have an opinions on this? From our conversations in London it seemed like you have some experience with profiling (or at least timing).

Avs uses python timeit module and some of IPython %timeit magic, so I will not consider this reliable/reproducible for timings less than 0.1 seconds. [1] [2]

< TLDR >
The TLDR is that you have to select what to measure: average execution time or "platonic" execution time (the fastest and usually not very representative). When you do this you need to understand how timeit and %timeit and other tools are measuring. If your individual executions are fast, there is ton of noise that affects the measurements.
< /TLDR >

If you don't want to limit the noise (which is very complicated), then another approach is to limit/understand the effect of the noise using statistics. As the initial distribution for the time measurements has skewness <0, assuming that the distribution is homeomorphic to P(X=x)=a e^(-a(x-b)) for X>b, 0 otherwise (where X is a random variable and b,a are real numbers), then the typical procedure to obtain stable benchmarks by limiting the noise is measuring execution n times, doing the mean of this values and repeating this r times. Using this fact, the distribution representing the means: Y=1/n(X1+X2+...) For the central limit theorem we know that P(X=Y) ~ N(sigma, mu) Where N is the normal distribution of stdv sigma and mean mu. This is a profound result and therefore you don't need to worry about the left skewness or higher central momenta of the distribution of errors, the means of the execution times will be always asymptotically a normal distribution. This means that reporting the mean and the standard deviation of a statistically significant number of experiments is enough because there are no higher momenta available. Notice that this is true even if the errors are on one side of the execution result. This is trying to answer another problem that is "what is the most representative execution time in this particular machine" as opposed to "what is the platonic execution time for this set of instructions". IMHO users want reproducibility and some sense of stability, even under noisy and unstable conditions.

I think it also depends on how seriously do you want this to be. A lot of people are using avs because they don't care about the inexactitudes that much, or because they don't know best. Actually, avs is doing a lot of good work to set up the environment and produce proper metrics (you can select warmup_time,repeat,number and sample times among other options) which I think you can use to be very close to stable benchmarking. Another set of things you probably want to do in the machine you run this is to deactivate frequency switching in the CPU that you are using and isolate one CPU core for running the benchmarks.

If you use avs you should try to keep the benchmarking functions as short as possible, leaving the measures and the control to avs itself.

@pablogsal I think the goal here is mostly going to be differential benchmarks. I think for now we can take the performance metric is "does set of instructions X perform faster than set of instructions Y", and the competing interests are (likely in order of prevalence):

A.) On the average machine, does it perform better in a tight loop
B.) On the average machine, does it perform better when high-latency is needed
C.) Does it perform better in resource-constrained environments.

I imagine that what we'll be doing with asv is trying to find a model that gives approximate answers to these questions given that it will be run on only one machine. For practical purposes, I think that we can easily accept errors that are on the order of magnitude of the natural variation of the project (e.g. if some machines perform faster with instructions X and some faster with instructions Y, or 1 STD of the variation in the differential performance on various common inputs).

Given imperfect tools to do this (and given that I am not prepared to maintain a separate benchmarking tool), I'd like to use the best configuration for asv (or another standard benchmarking tool if there's a better one) that optimizes these goals and lets me make actionable decisions about algorithm choices, even if the decision is "the difference between these is not statistically significant".

So, given that, do you have thoughts about whether we should write tests like:

def time_parse_iso(self):
    for n in range(1000):
        parse('2017-09-12')

So that we can add some averaging into the process and make it effectively "longer", or should we write tests like:

def time_parse_iso(self):
    parse('2017-09-12')

and trust that asv will figure out the best way to benchmark them for us?

If the main concern is the way that asv does things, probably it's best for us to open an issue and possibly a PR there.

@pganssle IMHO You should write test like this:

def time_parse_iso(self):
    parse('2017-09-12')

and configure warmup_time, repeat, number and sample times (in avs) accordingly to have enough samples and a good average.

If the main concern is the way that asv does things, probably it's best for us to open an issue and possibly a PR there.

I opened a couple last week, very wary of ...

and given that I am not prepared to maintain a separate benchmarking tool

The subprocess indirection asv uses makes it tough to extend the API (at least for one newbie), but I'm cautiously hopeful that some of those rough edges may get sanded down.

Is this part of the PoC or something you actually want merged?

PoC to demonstrate the asv continuous comparison. It should go into a separate PR-- it's a pretty nice perf improvement.

I'll revert this part in the next draft.

I think this is a pandas thing? Should we just drop this?

Good catch.