Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.90%. Comparing base (182504c) to head (7f71875).
⚠️ Report is 50 commits behind head on main.

@@            Coverage Diff             @@
##             main    #3574      +/-   ##
==========================================
+ Coverage   61.86%   61.90%   +0.03%     
==========================================
  Files          85       85              
  Lines       10137    10150      +13     
==========================================
+ Hits         6271     6283      +12     
- Misses       3866     3867       +1     

Not sure I fully grasp what is going on with codecov, it looks like the tests run and assert the things they should and use the funcionality promised.

In SpatialData we could also have a use case for this. Many users are not as experienced with the whole concept of sharding and chunking, so having the option to provide a maximum size with some explanation in documentation would be very beneficial.

I was curious how xarray handles sharding and from what I can tell it is also manually done there:

https://github.com/pydata/xarray/blob/6da8c2c92326d05b03bffa1a022d9f87f4f6278d/xarray/tests/test_backends.py#L2891-L2907

@jhamman Is this true? Would xarray benefit from a feature like this or are we thinking about the problem wrong and people should be doing this manually?

@ilan-gold I think that is correct at the moment. See also pydata/xarray#9947. In dask I am working on having the interface just provide a shard_factor argument. I think the interface mostly just depends on the user. Some users like to think more in sizes, particularly when they are lessed experienced with chunking / sharding as they can relate it easier to the capability of their hardware.

@d-v-b What is the take on the fact that this is pre-compression? Something I just realized messing around with this. Should we document that more clearly?

@d-v-b What is the take on the fact that this is pre-compression? Something I just realized messing around with this. Should we document that more clearly?

Maybe just indicate in the docs that this setting governs the in-memory size?

Also, how will this handle data types that don't have a computable size per element (like variable-length bytes)?

Maybe just indicate in the docs that this setting governs the in-memory size?

#3599

Also, how will this handle data types that don't have a computable size per element (like variable-length bytes)?

It takes in the same arguments as _auto_partition so itemsize is derived from the zarr data type when it inherits from HasItemSize as far as I can tell

The docs say for chunking:

If you are feeling lazy, you can let Zarr guess a chunk shape for your data by providing chunks='auto', although please note that the algorithm for guessing a chunk shape is based on simple heuristics and may be far from optimal

I could add an explanation of the heuristic for both sharding and chunking a bit here, or at least specify what to expect based on data type.