We've had a number of discussions around this, but I'm not aware of a consolidated issue. Currently our benchmarks and tests are mostly able to measure wall clock time. It would also be good to measure some other things, including

1. Average memory usage
2. Peak memory usage
3. Data spilled to disk
4. Timing data for compute, serialization, disk spilling

Some of these things are available on the dashboard and in performance reports, but those numbers are not very visible, and are not easy to extract in an automated way.

Ideally, whatever benchmarking tool we use would be able to include custom metrics like the above, thus making it easier to spot regressions in them (cf. #190). I suspect at least some of them could be instrumented with a lightweight scheduler plugin, along the lines of this one, which measures task group timing data. I might find something like this to be a useful thing when making plugins for this purpose.