Skip to content

Numcodecs Delta filter with astype not working with Zarr 3 #663

New issue
New issue
Closed
#664
Closed
Numcodecs Delta filter with astype not working with Zarr 3#663
#664
@rabernat

Description

@rabernat
rabernat
opened on Dec 1, 2024

Here's a simple reproducer

import zarr
from numcodecs.zarr3 import Delta

store = zarr.storage.MemoryStore()
array = zarr.create(
    store=store, shape=100, chunks=100, dtype="i8", zarr_format=3,
    codecs=[Delta(dtype="i8", astype="i2"), zarr.codecs.BytesCodec()]
)

# works
array[:] = 1

# errors
array[:]
# -> ValueError: cannot reshape array of size 25 into shape (100,)
traceback 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[29], [line 1](vscode-notebook-cell:?execution_count=29&line=1)
----> [1](vscode-notebook-cell:?execution_count=29&line=1) array[:]

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1953, in Array.__getitem__(self, selection)
   [1951](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1951)     return self.vindex[cast(CoordinateSelection | MaskSelection, selection)]
   [1952](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1952) elif is_pure_orthogonal_indexing(pure_selection, self.ndim):
-> [1953](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1953)     return self.get_orthogonal_selection(pure_selection, fields=fields)
   [1954](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1954) else:
   [1955](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1955)     return self.get_basic_selection(cast(BasicSelection, pure_selection), fields=fields)

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:43, in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     [41](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:41) extra_args = len(args) - len(all_args)
     [42](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:42) if extra_args <= 0:
---> [43](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:43)     return f(*args, **kwargs)
     [45](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:45) # extra_args > 0
     [46](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:46) args_msg = [
     [47](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:47)     f"{name}={arg}"
     [48](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:48)     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:], strict=False)
     [49](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/_compat.py:49) ]

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2395, in Array.get_orthogonal_selection(self, selection, out, fields, prototype)
   [2393](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2393)     prototype = default_buffer_prototype()
   [2394](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2394) indexer = OrthogonalIndexer(selection, self.shape, self.metadata.chunk_grid)
-> [2395](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2395) return sync(
   [2396](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2396)     self._async_array._get_selection(
   [2397](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2397)         indexer=indexer, out=out, fields=fields, prototype=prototype
   [2398](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2398)     )
   [2399](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:2399) )

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:141, in sync(coro, loop, timeout)
    [138](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:138) return_result = next(iter(finished)).result()
    [140](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:140) if isinstance(return_result, BaseException):
--> [141](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:141)     raise return_result
    [142](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:142) else:
    [143](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:143)     return return_result

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:100, in _runner(coro)
     [95](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:95) """
     [96](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:96) Await a coroutine and return the result of running it. If awaiting the coroutine raises an
     [97](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:97) exception, the exception will be returned.
     [98](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:98) """
     [99](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:99) try:
--> [100](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:100)     return await coro
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:101) except Exception as ex:
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/sync.py:102)     return ex

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1009, in AsyncArray._get_selection(self, indexer, prototype, out, fields)
   [1001](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1001)     out_buffer = prototype.nd_buffer.create(
   [1002](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1002)         shape=indexer.shape,
   [1003](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1003)         dtype=out_dtype,
   [1004](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1004)         order=self.order,
   [1005](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1005)         fill_value=self.metadata.fill_value,
   [1006](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1006)     )
   [1007](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1007) if product(indexer.shape) > 0:
   [1008](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1008)     # reading chunks and decoding them
-> [1009](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1009)     await self.codec_pipeline.read(
   [1010](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1010)         [
   [1011](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1011)             (
   [1012](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1012)                 self.store_path / self.metadata.encode_chunk_key(chunk_coords),
   [1013](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1013)                 self.metadata.get_chunk_spec(chunk_coords, self.order, prototype=prototype),
   [1014](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1014)                 chunk_selection,
   [1015](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1015)                 out_selection,
   [1016](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1016)             )
   [1017](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1017)             for chunk_coords, chunk_selection, out_selection in indexer
   [1018](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1018)         ],
   [1019](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1019)         out_buffer,
   [1020](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1020)         drop_axes=indexer.drop_axes,
   [1021](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1021)     )
   [1022](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/array.py:1022) return out_buffer.as_ndarray_like()

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:440, in BatchedCodecPipeline.read(self, batch_info, out, drop_axes)
    [434](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:434) async def read(
    [435](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:435)     self,
    [436](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:436)     batch_info: Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]],
    [437](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:437)     out: NDBuffer,
    [438](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:438)     drop_axes: tuple[int, ...] = (),
    [439](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:439) ) -> None:
--> [440](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:440)     await concurrent_map(
    [441](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:441)         [
    [442](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:442)             (single_batch_info, out, drop_axes)
    [443](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:443)             for single_batch_info in batched(batch_info, self.batch_size)
    [444](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:444)         ],
    [445](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:445)         self.read_batch,
    [446](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:446)         config.get("async.concurrency"),
    [447](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:447)     )

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:67, in concurrent_map(items, func, limit)
     [64](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:64)     async with sem:
     [65](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65)         return await func(*item)
---> [67](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:67) return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65, in concurrent_map.<locals>.run(item)
     [63](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:63) async def run(item: tuple[Any]) -> V:
     [64](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:64)     async with sem:
---> [65](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65)         return await func(*item)

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:270, in BatchedCodecPipeline.read_batch(self, batch_info, out, drop_axes)
    [261](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:261) else:
    [262](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:262)     chunk_bytes_batch = await concurrent_map(
    [263](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:263)         [
    [264](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:264)             (byte_getter, array_spec.prototype)
   (...)
    [268](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:268)         config.get("async.concurrency"),
    [269](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:269)     )
--> [270](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:270)     chunk_array_batch = await self.decode_batch(
    [271](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:271)         [
    [272](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:272)             (chunk_bytes, chunk_spec)
    [273](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:273)             for chunk_bytes, (_, chunk_spec, _, _) in zip(
    [274](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:274)                 chunk_bytes_batch, batch_info, strict=False
    [275](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:275)             )
    [276](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:276)         ],
    [277](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:277)     )
    [278](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:278)     for chunk_array, (_, chunk_spec, chunk_selection, out_selection) in zip(
    [279](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:279)         chunk_array_batch, batch_info, strict=False
    [280](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:280)     ):
    [281](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:281)         if chunk_array is not None:

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:177, in BatchedCodecPipeline.decode_batch(self, chunk_bytes_and_specs)
    [172](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:172)     chunk_bytes_batch = await bb_codec.decode(
    [173](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:173)         zip(chunk_bytes_batch, chunk_spec_batch, strict=False)
    [174](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:174)     )
    [176](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:176) ab_codec, chunk_spec_batch = ab_codec_with_spec
--> [177](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:177) chunk_array_batch = await ab_codec.decode(
    [178](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:178)     zip(chunk_bytes_batch, chunk_spec_batch, strict=False)
    [179](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:179) )
    [181](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:181) for aa_codec, chunk_spec_batch in aa_codecs_with_spec[::-1]:
    [182](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:182)     chunk_array_batch = await aa_codec.decode(
    [183](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:183)         zip(chunk_array_batch, chunk_spec_batch, strict=False)
    [184](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/codec_pipeline.py:184)     )

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:129, in BaseCodec.decode(self, chunks_and_specs)
    [113](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:113) async def decode(
    [114](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:114)     self,
    [115](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:115)     chunks_and_specs: Iterable[tuple[CodecOutput | None, ArraySpec]],
    [116](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:116) ) -> Iterable[CodecInput | None]:
    [117](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:117)     """Decodes a batch of chunks.
    [118](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:118)     Chunks can be None in which case they are ignored by the codec.
    [119](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:119) 
   (...)
    [127](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:127)     Iterable[CodecInput | None]
    [128](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:128)     """
--> [129](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:129)     return await _batching_helper(self._decode_single, chunks_and_specs)

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:407, in _batching_helper(func, batch_info)
    [403](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:403) async def _batching_helper(
    [404](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:404)     func: Callable[[CodecInput, ArraySpec], Awaitable[CodecOutput | None]],
    [405](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:405)     batch_info: Iterable[tuple[CodecInput | None, ArraySpec]],
    [406](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:406) ) -> list[CodecOutput | None]:
--> [407](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:407)     return await concurrent_map(
    [408](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:408)         list(batch_info),
    [409](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:409)         _noop_for_none(func),
    [410](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:410)         config.get("async.concurrency"),
    [411](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:411)     )

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:67, in concurrent_map(items, func, limit)
     [64](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:64)     async with sem:
     [65](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65)         return await func(*item)
---> [67](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:67) return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65, in concurrent_map.<locals>.run(item)
     [63](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:63) async def run(item: tuple[Any]) -> V:
     [64](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:64)     async with sem:
---> [65](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/common.py:65)         return await func(*item)

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:420, in _noop_for_none.<locals>.wrap(chunk, chunk_spec)
    [418](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:418) if chunk is None:
    [419](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:419)     return None
--> [420](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/abc/codec.py:420) return await func(chunk, chunk_spec)

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:94, in BytesCodec._decode_single(self, chunk_bytes, chunk_spec)
     [92](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:92) # ensure correct chunk shape
     [93](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:93) if chunk_array.shape != chunk_spec.shape:
---> [94](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:94)     chunk_array = chunk_array.reshape(
     [95](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:95)         chunk_spec.shape,
     [96](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:96)     )
     [97](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/codecs/bytes.py:97) return chunk_array

File ~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/buffer/core.py:441, in NDBuffer.reshape(self, newshape)
    [440](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/buffer/core.py:440) def reshape(self, newshape: ChunkCoords | Literal[-1]) -> Self:
--> [441](https://file+.vscode-resource.vscode-cdn.net/Users/rabernat/gh/earth-mover/arraylake/scripts/~/mambaforge/envs/arraylake-local/lib/python3.11/site-packages/zarr/core/buffer/core.py:441)     return self.__class__(self._data.reshape(newshape))

ValueError: cannot reshape array of size 25 into shape (100,)

Since the Delta filter can change the item size, I believe that this one-liner

numcodecs/numcodecs/zarr3.py

Line 269 in f43449b

Delta = _add_docstring(_make_array_array_codec("delta", "Delta"), "numcodecs.delta.Delta")

needs to be something more like this:

numcodecs/numcodecs/zarr3.py

Lines 325 to 339 in f43449b

@_add_docstring_wrapper("numcodecs.astype.AsType")
class AsType(_NumcodecsArrayArrayCodec):
codec_name = f"{CODEC_PREFIX}astype"
def __init__(self, **codec_config: JSON) -> None:
super().__init__(**codec_config)
def resolve_metadata(self, chunk_spec: ArraySpec) -> ArraySpec:
return replace(chunk_spec, dtype=np.dtype(self.codec_config["encode_dtype"])) # type: ignore[arg-type]
def evolve_from_array_spec(self, array_spec: ArraySpec) -> AsType:
decode_dtype = self.codec_config.get("decode_dtype")
if str(array_spec.dtype) != decode_dtype:
return AsType(**{**self.codec_config, "decode_dtype": str(array_spec.dtype)})
return self

What do you think @normanrz?

Also makes me wish we had more thorough testing of these adaptors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions