@maxrjones I ran my workflow and still got gibberish: https://nbviewer.org/gist/rsignell/32f563cfbe83c20ac01aba00190e5912
Did I need to update some other stuff in addition to virtualizarr?
Or perhaps more likely I did something else wrong. 😕

@maxrjones I ran my workflow and still got gibberish: nbviewer.org/gist/rsignell/32f563cfbe83c20ac01aba00190e5912 Did I need to update some other stuff in addition to virtualizarr? Or perhaps more likely I did something else wrong. 😕

Yeah, this fix is only for Icechunk. Kerchunk uses Zarr specification 2, which will require separate fix or adapting our Kerchunk writer to use Zarr format 3. I'd be tempted to go with the second option because converting from Zarr V3 to V2 for Kerchunk adds a lot of surface area for potential bugs, but that would be a separate PR.

@maxrjones cool, I wanted to use icechunk anyway! And as you said, it does solve my use case:
https://nbviewer.org/gist/rsignell/480cf6ac5142ce2b828199bcc601a93e

So to check my understanding, the reason this failed was because the default array->bytes codec (i.e. the "serializer") in zarr v3 terms was for little endian? Which is fine until you use it to decode big-endian data, when it will give you gibberish?

That's right.

We still likely have bugs because a user can define their own codec that doesn't subclass from Zarr python. In that case, their custom codec will be dropped by

I expect a lot of our codec handling will get much easier with @d-v-b's refactor in Zarr-Python, which is why I have been proposing small patch fixes rather than more fundamental changes.

their custom codec will be dropped by

Could we at least raise in that scenario?

their custom codec will be dropped by

Could we at least raise in that scenario?

yeah we definitely should raise in that scenario.