In the current state this might introduce variance in the code we measure since we'll also measure the pytest hook.
I think it would be better to do it the other way around: have pytest_runtest_call call _run_with_instrumentation

A way to do it might be to and override runtest but don't hesitate if you have another idea!

After digging some more I found we can use pytest_pyfunc_call instead of pytest_runtest_protocol (runtest invokes this hook, see https://github.com/pytest-dev/pytest/blob/5bb1363435a8cb3e2010505dbeb1e015c36beed6/src/_pytest/python.py#L1762-L1764)

I got the tests to pass locally but I may be missing something so let me know if you don't think this is the right thing to do

When used with the fixture, the fixture payload shouldn't be executed wrapped again within pyfunc_call since it's already wrapped by the test itself.

for example, this would probably fail because of the warmup:

# This doesn't have any pytest marker but defines a bench since its using the fixture
def test_bench(benchmark):
    # ... some preprocessing
    called_once = False

    @benchmark
    def _():
        nonlocal called_once
        if not called_once:
            called_once = True
        else:
            raise Exception("bench codewas called twice but actual bench context only once")

Don't hesitate to add that as an additional test

I started to get really confused with the intention here since the above example also fails with master, see #30

I suspect we need to do something similar to what pytest-rerunfailures does to clear the cached results between the cache priming run and the instrumented run: https://github.com/pytest-dev/pytest-rerunfailures/blob/a53b9344c0d7a491a3cc53d91c7319696651d21b/src/pytest_rerunfailures.py#L565-L567

Love this refactor!
However, when a benchmark is defined from a fixture, we should still perform the warmup at this level and not in the benchmark fixture(see the other comment I left on it).

I think the warmup shouldn't be included in the wrapper but handled at the protocol level

Ah, this also explains the issue we ran into on our own trying to use tmp_path in benchmark tests, e.g.:

@pytest.mark.benchmark
def test_tmp_path_benchmark(tmp_path: Path):
    tmp_path.mkdir()

since you introduced this try block, can you add a test bench raising an exception?