This issue is just to make sure that benchmarks compile and run with each commit. This is not about tracking perf regressions.

The motivation is that throughout the history the benches have been multiple times broken, mostly for stupid reasons (e.g. hardcoded file paths didn't survive file renamings and so on). Regardless of whether we'll eventually have some perf tracking mechanism, we should at least have a working local benchmarking environment so that contributors can easily microbench their PRs and don't have to context switch for making things compile (and push irrelevant changes thereby).