not sure if adding the blosc dep here is correct, since we are vendoring it (cc @jakirkham)

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.33%. Comparing base (dbcf37b) to head (9459290).
⚠️ Report is 20 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (dbcf37b) and HEAD (9459290). Click for more details.

HEAD has 10 uploads less than BASE

Flag	BASE (dbcf37b)	HEAD (9459290)
	15	5

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   99.82%   90.33%   -9.49%     
==========================================
  Files          64       64              
  Lines        2804     2804              
==========================================
- Hits         2799     2533     -266     
- Misses          5      271     +266     

cc @mkitti in case you have any advice

not sure if adding the blosc dep here is correct, since we are vendoring it (cc @jakirkham)

I'll just point out that blosc is not even a dependency of the numcodecs package in conda-forge:
https://github.com/conda-forge/numcodecs-feedstock/blob/main/recipe/meta.yaml

I think this situation is a very weird. We're not even modifying the vendored blosc build here.

In contrast, the blosc package in conda-forge depends on lz4-c, snappy, zlib, and zstd packages, as this package should.
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/meta.yaml

That said, I have no idea how one would have those dependencies in PyPI via wheels.

cramjam, anyone?? This would just be the reason for them to solidify the blosc backend, but they have all the others in a statically-linked wheel

statically-linked wheel

Is a big statically-linked wheel the ideal here?

Is a big statically-linked wheel the ideal here?

If you're worried about dependencies, packaging and how to link to system libs, I suggest that yes. I note that cramjam is ~2MB. Also, zarr likes rust crates, and cramjam can be called directly from rust, and also return python-consumable buffers.

I took a look at libcramjam. I can already see several major issues. For example, I do not see checksum options for Zstandard. Now in certain circumstances, perhaps this could be fine if we had a plugin system that allowed for multiple implementations of the same codec.

I think I would much rather have a direct dependency on libzstd rather than going through another intermediary.

Rather than

pixi run pip install -v -e '.[test,test_extras,msgpack,crc32c,pcodec,zfpy]'

I think either pixi install or pixi reinstall should be sufficient. Pixi should be able to handle all the dependencies. If the optional dependencies are desired, they could be inserted via features and (sub)environments.

pixi reinstall is relatively recent.

I noticed the config has an editable install, just be aware there are two open Pixi issues about editable installs causing lockfile churn:

• Remove editable from the lock-file prefix-dev/pixi#1274
• Leave out development package out of lock file prefix-dev/pixi#2512

the editable install is not at all required, feel free to amend that if it makes things easier. i'm far from a pixi expert so I'm happy to defer to others here!

I noticed the config has an editable install, just be aware there are two open Pixi issues about editable installs causing lockfile churn.

Why is this a problem? For a library like this, we could just add the lock file to .gitignore.

@mkitti do you think we need anything more in this PR?

Add this the package as an editable install, remove the pip dependency, and do not recommend the use of pixi run pip install -e . Use pixi reinstall instead.

this was handled by #766