Ok, thanks @GenevieveBuckley and @jakirkham. A starting point for a minimal example can be the following, which captures one of the failing tests with for a function defined in napari. We can probably simplify it before posting, but I want to start with the full thing to make sure we're capturing the problem.

import dask.array as da
import numpy as np
import time


def calc_data_range(data):
    """Calculate range of data values. If all values are equal return [0, 1].

    Parameters
    ----------
    data : array
        Data to calculate range of values over.

    Returns
    -------
    values : list of float
        Range of values.

    Notes
    -----
    If the data type is uint8, no calculation is performed, and 0-255 is
    returned.
    """
    if data.dtype == np.uint8:
        return [0, 255]
    if np.prod(data.shape) > 1e6:
        # If data is very large take the average of the top, bottom, and
        # middle slices
        bottom_plane_idx = (0,) * (data.ndim - 2)
        middle_plane_idx = tuple(s // 2 for s in data.shape[:-2])
        top_plane_idx = tuple(s - 1 for s in data.shape[:-2])
        idxs = [bottom_plane_idx, middle_plane_idx, top_plane_idx]
        reduced_data = [
            [np.max(data[idx]) for idx in idxs],
            [np.min(data[idx]) for idx in idxs],
        ]
    else:
        reduced_data = data

    min_val = np.min(reduced_data)
    max_val = np.max(reduced_data)

    if min_val == max_val:
        min_val = 0
        max_val = 1
    return [float(min_val), float(max_val)]


data = da.random.random(
    size=(100_000, 1000, 1000), chunks=(1, 1000, 1000)
)

t0 = time.time()
val = calc_data_range(data)
t1 = time.time()
print(t1 - t0)

Which with dask 2.27.0 takes ~0.5 seconds and with dask 2.28.0 takes ~15 seconds. Our test has a 2 second timeout which causes it to now fail.

I'll see if I can clean this up/ simplify it a bit too

Uh oh!

failing dask test #1656

Description

🐛 Bug

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions