Skip to content

Caching exceptions seems to interfer with pytests traceback cleaning #10363

Closed
Closed
Caching exceptions seems to interfer with pytests traceback cleaning#10363
@pmeier

Description

@pmeier
pmeier
opened on Oct 10, 2022

This was originally reported in #10362 and was deemed "probably [...] a pytest bug" by @RonnyPfannschmidt.

We have tests that are paremetrized with two values. Let's call them foo and bar for this. Every foo has multiple bar's associated with it. Our tests look something like this:

@pytest.mark.parametrize(("foo", "bar"), ...)
def test_foo_bar(foo, bar):
    processed_foo = do_something_expensive_with_foo(foo)

    do_something_with_foo_and_bar(processed_foo, bar)

This means do_something_expensive_with_foo will be hit a lot with the same inputs and thus we want to cache it. Since the call can fail and we want to cache nevertheless, we have our own @cache decorator that is able to do that:

def cache(fn):
    """Similar to :func:`functools.cache` (Python >= 3.8) or :func:`functools.lru_cache` with infinite cache size,
    but this also caches exceptions.
    """
    sentinel = object()
    out_cache = {}
    exc_cache = {}

    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        key = args + tuple(kwargs.values())

        out = out_cache.get(key, sentinel)
        if out is not sentinel:
            return out

        exc = exc_cache.get(key, sentinel)
        if exc is not sentinel:
            raise exc

        try:
            out = fn(*args, **kwargs)
        except Exception as exc:
            exc_cache[key] = exc
            raise exc

        out_cache[key] = out
        return out

    return wrapper

It works exactly as intended except for one thing: if the call fails, it creates traceback abominations like this:

Traceback 

Traceback (most recent call last):
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 225, in wrapper
    raise exc
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 259, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    raise e
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 1761, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 225, in wrapper
    raise exc
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 259, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    raise e
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 1761, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 225, in wrapper
    raise exc
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 259, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    raise e
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 1761, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 225, in wrapper
    raise exc
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 259, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    raise e
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 1761, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 225, in wrapper
    raise exc
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 259, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    raise e
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 1761, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 231, in wrapper
    raise exc
  File "/home/philip/git/pytorch/torchvision/test/common_utils.py", line 228, in wrapper
    out = fn(*args, **kwargs)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 24, in script
    return torch.jit.script(fn)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/torch/jit/_script.py", line 1344, in script
    qualified_name, ast, _rcb, get_default_args(obj)
RuntimeError: 
Ellipses followed by tensor indexing is currently not supported:
  File "/home/philip/git/pytorch/torchvision/torchvision/prototype/transforms/functional/_geometry.py", line 53
    )

    bounding_box[..., [0, 2]] = image_size[1] - bounding_box[..., [2, 0]]
                                                ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

    return convert_format_bounding_box(

As you can see, the largest part of the traceback is coming from pytest and has nothing to do with the actual failure. Removing the @cache decorator from do_something_expensive_with_foo gives a normal traceback:

Traceback 

Traceback (most recent call last):
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 91, in test_scripted_vs_eager
    kernel_scripted = script(kernel_eager)
  File "/home/philip/git/pytorch/torchvision/test/test_prototype_transforms_functional.py", line 24, in script
    return torch.jit.script(fn)
  File "/home/philip/.local/opt/mambaforge/envs/torchvision-dev/lib/python3.7/site-packages/torch/jit/_script.py", line 1344, in script
    qualified_name, ast, _rcb, get_default_args(obj)
RuntimeError: 
Ellipses followed by tensor indexing is currently not supported:
  File "/home/philip/git/pytorch/torchvision/torchvision/prototype/transforms/functional/_geometry.py", line 53
    )

    bounding_box[..., [0, 2]] = image_size[1] - bounding_box[..., [2, 0]]
                                                ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

    return convert_format_bounding_box(

I'm guessing pytest removes all the internal calls from the traceback here. Caching exceptions seem to interfer with that.

This happens on the latest pytest==7.1.3. We are running with --tb=native, but I'm guessing this plays no role here, since the caching seems to be the root cause.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions