Skip to content

Replace with cluster context manager with pytest fixture #1966

@mrocklin

Description

@mrocklin

As discussed in the testing documentation many of our tests follow the following pattern:

def test_submit_sync(loop):
    with cluster() as (s, [a, b]):
        with Client(('127.0.0.1', s['port']), loop=loop) as c:
            future = c.submit(inc, 1)
            assert future.result() == 2

            a['proc'].terminate()  # sometimes interact with the scheduler or worker processes

This pattern is repeated frequently throughout tests and adds to the development learning curve when first starting out. It would be nice to refactor it to be a pytest fixture instead

def test_submit_sync(cluster, c):
    future = c.submit(inc, 1)
    assert future.result() == 2

    cluster['workers'][0]['proc'].terminate()

This would make things much cleaner.

The old context manager needs to still work for two reasons:

  1. It is used in downstream dependencies, like xarray
  2. It is used in our own tests sometimes with extra configuration

This is a relatively simple but tedious task. Most of the work here is rewriting most of the test suite to remove the repetitive lines. If anyone has some time during a long flight coming up it would be welcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions