Zarr LUTs#787
Conversation
|
There are presently two versions of the zarr implementation:
The first worked well, the second had some issues with fill values. Zarr by default sets the fill value of an array to 0, whereas xarray sets it to |
|
Some performance results: Top graph is zarr, bottom is netcdf. Larger 5GB LUT generation: |
…r zarr or netcdf; removed force options
|
Closing as this has been integrated separately in ISOFIT v4.0.0 |
Implements the ability to create Zarr stores instead of NetCDF when creating a LUT.
Createclass was generalized to be subclassable, introducing:CreateNetCDFcurrent implementationCreateZarrnew Zarr-focused versionxarrayheavily but we could look into using thezarrpackage itself, optionallyluts.createis the primary entry point to determine which Create class to returnluts.loadupdated to detect zarrSome further testing of the efficiency of the xarray implementation is still needed. At the moment, when
flushis called, each variable for each point in the queued points is written to disk consecutively. This may be fine because I target the specific region in the Zarr store to write to for each, but this needs to be tested on larger luts.