In python/importlib_metadata#294, I'm employing pyperf to measure the performance difference between the locally-checked out code and the main branch. To my dismay, pyperf reports significant variance between the runs. By design, the commands run are identical and the python executable is identical. The only difference is which copy of the code is installed into each environment (but the code is identical since the PR is based on what was main at the time and there are no code changes).

One variable is that since both commands are run in .tox, the first run will see ./perf in the current directory and the second run will see ./perf and ./perf-ref in the current directory, but that seems unlikely to affect the performance of metadata handling.

Historically, I've had good reliability with simply python -m timeit. I expected to get similar reliability from pyperf.

You can replicate the results by checking out the project and running tox -e 'perf{,-ref}'.

I'm really excited about the prospect of relying on pyperf to provide the comparison reports of main against local, but right now my optimism is dashed.

I skimmed through the tuning docs, but they seem highly platform-specific and doesn't explain why pyperf timeit is getting jittery results where timeit does not.

It's quite possible, even likely, that I'm missing something obvious, so I humbly ask for advice. Is there any easy way to make the performance results more stable in a cross-platform way?