I just watched Michael Droettboom's talk on his excellent and excellently named project, Airspeed Velocity, which tracks a project's benchmarks over time to ensure that benchmarks don't degrade. It seems pretty bloody awesome and I definitely think we should use it, though the logistics will be a bit tricky.

Talk link:
http://youtu.be/OsxJ5O6h8s0

Once we figure out where to run this, we can run 0.7 - 0.10 and then run master or each PR. (As Michael mentions in his talk, the PR stuff needs to be manual after code vetting or it will be a security risk. Running on a VM messes with the benchmarks.) (Obviously it will be a security risk even after vetting, but a much smaller one. =)