Skip to content

Chunks error raised whilst regridding #665

@davidhassell

Description

@davidhassell

Foe version 3.14.0 to 3.15.1, regrid operations with Field.regrids or Field.regridc whose regrid axes are chunked with more than one chunk will fail as follows:

>>> obs.data.chunks   # axes 1 and 2 are the regrid  axes
((322, 322, 322, 322, 164), (322, 38), (322, 322, 76)) 
>>> obs_regrid = obs.regrids(model, method="linear")
File ~/miniconda3/lib/python3.10/site-packages/dask/array/rechunk.py:307, in rechunk(x, chunks, threshold, block_size_limit, balance, method)
    305 if isinstance(chunks, (tuple, list)):
    306     chunks = tuple(lc if lc is not None else rc for lc, rc in zip(chunks, x.chunks))
--> 307 chunks = normalize_chunks(
    308     chunks, x.shape, limit=block_size_limit, dtype=x.dtype, previous_chunks=x.chunks
    309 )
    311 # Now chunks are tuple of tuples
    312 ndim = x.ndim

File ~/miniconda3/lib/python3.10/site-packages/dask/array/core.py:3128, in normalize_chunks(chunks, shape, limit, dtype, previous_chunks)
   3120         raise ValueError(
   3121             "Input array has %d dimensions but the supplied "
   3122             "chunks has only %d dimensions" % (len(shape), len(chunks))
   3123         )
   3124     if not all(
   3125         c == s or (math.isnan(c) or math.isnan(s))
   3126         for c, s in zip(map(sum, chunks), shape)
   3127     ):
-> 3128         raise ValueError(
   3129             "Chunks do not add up to shape. "
   3130             "Got chunks=%s, shape=%s" % (chunks, shape)
   3131         )
   3133 return tuple(
   3134     tuple(int(x) if not math.isnan(x) else np.nan for x in c) for c in chunks
   3135 )

ValueError: Chunks do not add up to shape. Got chunks=((322, 322, 322, 322, 164), (-1,), (-1,)), shape=(1452, 360, 720)

The problem lies in cf.Data._regrid rechunking the regrid axes to, (-1,) as opposed to the correct -1.

PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingregriddingRelating to regridding operations

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions