Skip to content

Replace custom Sage unit test discovery by pytest #30738

Open
Open
Replace custom Sage unit test discovery by pytest#30738
Labels
c: doctest frameworkp: major / 3s: needs workt: enhancementt: refactoring
@tobiasdiez

Description

@tobiasdiez
tobiasdiez
opened on Oct 7, 2020

Currently, Sage code contains _test_xyz methods that are invoked via doctests and discovered via a custom TestSuite class. This ticket is to replace this custom test discovery by the use of the established pytest framework https://docs.pytest.org/ (this is in line with #28936 - using mainstream Python test infrastructure).

Advantages of this approach:

  • Reuse standard test infrastructure instead of a custom test discovery interface (this is a big maintenance relief, especially given the large list of todos in the TestSuite code).
  • Better error messages in case a test fails.
  • Easier onboarding of new Python developers.
  • Clear separation of test methods vs production code. For example, the *_test.py files can easily be excluded from distribution.
  • Good support by IDEs (auto completion, test discovery and easier invocation). For example, VSCode automatically discovers all pytest tests, provides a convenient interface to run them (all, a selection, only failed ones) and allows to directly start a debug session for a failing test. See https://code.visualstudio.com/docs/python/testing
  • Less and clearer code (no hidden assumptions).

Disadvantages of this approach:

  • Current developers have to learn pytest and its conventions (files must be named *_test.py and test methods must start with test_).
  • Tests no longer live in the same file as the code (I would call this an advantage, but some people may prefer to have them really close together).

In order to migrate, the following changes are necessary:

  • Move _test_xyz from a category module to new file <module>_test.py (e.g. finite_semigroups_test.py).
  • Rewrite tests using pytest (which is straightforward and more or less only amounts to removing all the custom code involving self_tester).
  • Create a new test class for the parent class under test (e.g TestFiniteSemigroup).
  • Create a static method category_instances() in this test class returning a list of instances to test.

For the moment, I did this only for a few test methods in sets_cat using finite semigroups as example to illustrate the process.

The new implementation is as follows:

  • The test class for the category (e.g. in sets_cat_test.py) defines generic tests that should pass for every object in that category. Moreover, the test class may provides instances (objects) to test via the method category_instances.
  • Running pytest finds the test file finite_semigroups_test.py, then determines the categories of the objects (returned from category_instances) and adds all generic test methods for that category to the test class (this happens in conftest.py).
  • Each test method can have the parameter category_instance which is bound to the return value of the category_instances method.

Running cd src && pytest --verbose (from a virtual env with sage and pytest installed), the output is as follows:

sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                         [  2%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                    [  4%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                               [  6%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                        [  8%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                   [ 10%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                              [ 13%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                 [ 15%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                            [ 17%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                       [ 19%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                                   [ 21%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                              [ 23%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                         [ 26%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                                      [ 28%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                                 [ 30%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                            [ 32%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                           [ 34%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                      [ 36%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                 [ 39%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                         [ 41%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                    [ 43%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                               [ 45%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                                    [ 47%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                               [ 50%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                          [ 52%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                            [ 54%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                       [ 56%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                  [ 58%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                           [ 60%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                      [ 63%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                 [ 65%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                          [ 67%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                     [ 69%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                [ 71%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED                                                                    [ 73%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED                                                               [ 76%]
sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED                                                          [ 78%]
...

As one can see, each general test method is invoked for the three examples.

TODO (in follow-up tickets):

  • Complete migration to pytest.
  • Once that is done, stop invoking TestSuite in doctests and remove the TestSuite class.

See also:

CC: @mkoeppe @tscrim @nthiery @slel

Component: refactoring

Keywords: testsuite

Author: Tobias Diez

Branch/Commit: public/refactoring/pytest @ b047abe

Issue created by migration from https://trac.sagemath.org/ticket/30738

Metadata

Metadata

Assignees

No one assigned

    Labels

    c: doctest frameworkp: major / 3s: needs workt: enhancementt: refactoring

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions