The way that I typically "test" the fact that it is actually releasing the GIL is with this snippet from pandas (although more recently I was using dask's array.map_blocks, but the below is without a dependency on dask):

# from pandas.util.testing
from functools import wraps

def test_parallel(num_threads=2, kwargs_list=None):
    """Decorator to run the same function multiple times in parallel.

    Parameters
    ----------
    num_threads : int, optional
        The number of times the function is run in parallel.
    kwargs_list : list of dicts, optional
        The list of kwargs to update original
        function kwargs on different threads.
    Notes
    -----
    This decorator does not pass the return value of the decorated function.

    Original from scikit-image:

    https://github.com/scikit-image/scikit-image/pull/1519

    """

    assert num_threads > 0
    has_kwargs_list = kwargs_list is not None
    if has_kwargs_list:
        assert len(kwargs_list) == num_threads
    import threading

    def wrapper(func):
        @wraps(func)
        def inner(*args, **kwargs):
            if has_kwargs_list:
                update_kwargs = lambda i: dict(kwargs, **kwargs_list[i])
            else:
                update_kwargs = lambda i: kwargs
            threads = []
            for i in range(num_threads):
                updated_kwargs = update_kwargs(i)
                thread = threading.Thread(target=func, args=args, kwargs=updated_kwargs)
                threads.append(thread)
            for thread in threads:
                thread.start()
            for thread in threads:
                thread.join()

        return inner

    return wrapper

And with that function, you can then do:

arr = pygeos.polygons(np.random.rand(100_000, 4, 2))

# Running this 4 times sequentially:
def f():
    for i in range(4):
        pygeos.area(arr)

%timeit f()

# Running this 4 times but in parallel:
@test_parallel(4)
def g():
    pygeos.area(arr)

%timeit g()

and this way ensure that releasing the GIL is actually working / beneficial.

Now the above might be more for performance, and a way that we can include this in the asv benchmarks (that is where pandas also uses this, eg https://github.com/pandas-dev/pandas/blob/a4e19fa521221a30747bda1e0e72768e071f6d17/asv_bench/benchmarks/gil.py#L272-L301).
And the other kind of "test" you might be meaning here is simply to ensure correctness of the results? The function above could be adapted a bit to also collect the results, so we can ensure that serial vs parallel gives exactly the same.

TEST: Add multithreading tests for geometry-creating operations #164

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions