Currently PyTorch uses the builtin unittest module for testing. Would it be possible to add a dependency on pytest so developers can more easily write parametrized tests?

While working on test/test_distributions.py, @neerajprad, @alicanb, and @fritzo have found it challenging to write heavily parametrized tests. As a result, test coverage suffers. By contrast in Pyro we use heavily parametrized tests using pytest, and our Pyro tests seem to catch many bugs that aren't caught in PyTorch's own tests (#9917, #9521, #9977, #10241).

In particular, I'd like to be able to gather xfailing tests in batch

DISTRIBUTIONS = [Bernoulli, Beta, Cauchy, ..., Weibull]

@pytest.mark.parametrize('Dist', DISTRIBUTIONS)
def test_that_sometime_fails(Dist):
    dist = Dist(...)
    assert something(dist)

and then mark xfailing parameters

DISTRIBUTIONS = [
    Bernoulli,
    Beta,
    pytest.param(Cauchy, marks=[pytest.mark.xfail(reason='schema not found for node')]),
    ...
    Weibull,
]

pytest makes it easy to collect xfailing tests in batch (pytest -v --tb=no) and to see the entire list of xfailing tests, and makes it easy to run xfailing tests to see which have started passing since the last time tests were run (pytest -v, or pytest --runxfail). In contrast, unittest fixtures typically parametrize via for loops and can report only a single failure on each run. (Apologies if I'm unaware of convenient functionality in unittest!)

I think allowing usage of pytest in test/*.py could help improve PyTorch test coverage.

cc @mruberry