Stepwise is supposed to behave like -x but this doesn't work with xdist:
You may think of it as a combination of the -x option, which exits the test run after a failing test, and the --lf option from pytest-cache, which only runs failing tests.
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert False
def test_4():
assert True
def test_5():
assert True
def test_6():
assert TrueRunning it the first time, the test suite doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 5 passed in 0.23 seconds ====================================================================================Running it the second time, the test suite starts from the first failing test
but doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 3 passed in 0.21 seconds ====================================================================================Ok lets make the test pass:
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert True
def test_4():
assert True
def test_5():
assert True
def test_6():
assert Trueok it starts from the failure and continues the suite, this is correct
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 4 passed in 0.22 seconds =========================================================================================
and when run again the whole suite passes, this is correct too
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 6 passed in 0.21 seconds =========================================================================================
Stepwise is supposed to behave like
-xbut this doesn't work with xdist:Running it the first time, the test suite doesn't stop: (no -x behavior)
Running it the second time, the test suite starts from the first failing test
but doesn't stop: (no -x behavior)
Ok lets make the test pass:
ok it starts from the failure and continues the suite, this is correct
and when run again the whole suite passes, this is correct too