Skip to content

Remove unnecessary dtype conversion in icechunk writer#760

Merged
maxrjones merged 4 commits intozarr-developers:mainfrom
popododo0720:fix-unnecessary-dtype-conversion
Sep 3, 2025
Merged

Remove unnecessary dtype conversion in icechunk writer#760
maxrjones merged 4 commits intozarr-developers:mainfrom
popododo0720:fix-unnecessary-dtype-conversion

Conversation

@popododo0720
Copy link
Contributor

@popododo0720 popododo0720 commented Aug 8, 2025

Description

Removed unnecessary .to_native_dtype() conversion in icechunk writer as suggested by @d-v-b in #758.
Now passing metadata.data_type directly.

Changes

  • Simplified dtype parameter in VirtualiZarr/virtualizarr/writers/icechunk.py line 347

This is my first contribution!

@TomNicholas
Copy link
Member

@d-v-b your suggestion led our new contributor astray!

I think the error here might be actually indicative of an issue in zarr-python. I notice that the dtype argument in AsyncGroup.create_array() has type zarr.core.dtype.ZDTypeLike, but the dtype argument in AsyncGroup.require_array() has type numpy.typing.DTypeLike. Is that inconsistency deliberate?

@d-v-b
Copy link

d-v-b commented Aug 13, 2025

yes that's a zarr-python bug -- it looks like Group.require_array didn't get the full new dtypes makeover treatment

see zarr-developers/zarr-python#3377

Copy link
Member

@maxrjones maxrjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @popododo0720!

@TomNicholas
Copy link
Member

I'm confused what fixed this?

@maxrjones
Copy link
Member

I'm confused what fixed this?

#765 fixed the failing test. We were previously testing for a scenario in which the ArrayV3Metadata contains a numpy dtype, which should not occur since 3.1.0 and the use of ZDtype in the ArrayV3Metadata structure.

@maxrjones maxrjones merged commit 4c3c4fd into zarr-developers:main Sep 3, 2025
13 checks passed
TomNicholas added a commit that referenced this pull request Sep 24, 2025
maxrjones pushed a commit that referenced this pull request Nov 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unnecessary conversion back-and-forth between zarr data types and numpy dtypes

4 participants