So I'm thinking about generating the benchmarks instead of writing each of them. I see several possibilities for that:

• turning every gallery example into a benchmark. The nice thing is that examples correspond to relevant usecases. The downside is that we probably don't want the plotting stuff in the benchmark.
• do some inference on input type and run functions on random values, like in the benchmarking scripts I wrote for the sprint.

What do you think?

@emmanuelle yes this is a very good plan, but now the autogeneration should somehow fit into the asv benchmark format, this is why I pushed things here. =)

Codecov Report

Merging #3137 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3137   +/-   ##
=======================================
  Coverage   86.08%   86.08%           
=======================================
  Files         338      338           
  Lines       27234    27234           
=======================================
  Hits        23445    23445           
  Misses       3789     3789

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f2a517...c97f2b4. Read the comment docs.

Once this is merged, we'll add an entry to https://github.com/TomAugspurger/asv-runner/blob/master/tests/full.yml and it'll start running automatically.

@scikit-image/core I suggest merging this and allowing (/forcing) subsequent PRs to add benchmarks.

Perhaps we should add a section to the development guide about writing asv benchmarks?

At any rate I'm excited to have this merged! ;)

@TomAugspurger what happens when benchmarks themselves are added/removed/changed? Can one re-run a whole range of commits on benchmarks committed at the end?

@TomAugspurger You are asking for alot. I'm not sure it is possible to build anything older than 0.14 and have the tests pass ;)

@hmaarrfk tests don't need to pass, only benchmarks.

@jni could you, please, add a link from the comment above to somewhere (including the release_dev.rst)? I assume, both the contributors and the users might find it useful. Otherwise, lgtm. Thanks!

@jni if you add the comment requested by @soupault I will merge :-)

@emmanuelle @soupault done! =)

@TomAugspurger how does asv work with PRs? The specific feature that I'm interested in is the ability to run even a new benchmark (provided by the PR) on the master branch and the PR tip (or the merge commit) before merging. Is this easy enough to do?

@TomAugspurger that sounds like a fun challenge. And @Carreau has done the hard work with https://github.com/MeeseeksBox/MeeseeksDev. Can you give me a bit of background about what the bot would need to do? ie if I have a head commit and the master branch it split from, how do I:

• grab the benchmarks and asv conf from the head commit
• run the benchmarks on both
• publish the results somewhere

?

@jni As far as running goes, yes I think you've got it. You should be able to clone the PR and run asv continuous -f 1.1 HEAD master and that'll give you a text output that can be published on the issue. The bot could optionally take a -b flag to limit it to a certain set of benchmarks.

I think the main concern is scheduling these so that we aren't running these on-demand runs with

a.) eachother
b.) the nightly benchmarks

The benchmark machine is using Airflow right now to do the scheduling. Airflow supports external triggers, so this should be possible. I can help out with the airflow stuff if you want.

@emmanuelle if we merge this soon we can beat the 4th anniversary of #1061. =P

@jni

@hmaarrfk I know 😂 😭 😂

Thank you, Juan!

@jni good job on the asv PR.

For those looking to install asv locally, we need to wait either for 0.2.2 or install with pip and git

pip install git+https://github.com/airspeed-velocity/asv.git#egg=asv

@TomAugspurger none of the benchmarks added after this PR appear on the ASV server! Any ideas why? I looked at benchmarks/__init__.py and asv.conf.json but there's nothing there to suggest that new benchmarks shouldn't be run. Is there some asv cache that needs clearing when new benchmarks are added?

http://pandas.pydata.org/speed/scikit-image/#/

I think my 2 year old hit the power button on the benchmark machine, and I didn't notice till last night 😆 It should be catching up now, and we've had a talk about touching Daddy's stuff.

@TomAugspurger I feel like this is a longer term problem of not downloading from master. New benchmarks were added weeks ago...

Oh although I just reread your statement that you think it's a longer term problem on your end also. =P Ok let's see how we go... =)

Found the problem. When we initially set this up, I manually made the change adding a benchmarks/__init__.py on the ASV machine. But I never cleared that change, so the git pull wasn't working for scikit-image. Fixed that, and am re-running.

🎉 thanks @TomAugspurger!