Skip to content

Allow the HDF5 "chunk cache" to be set from cf.write #871

@davidhassell

Description

@davidhassell

netCDF4.createVariable has a chunk_cache keyword that allows for the setting of the amount of memory (in bytes) used in each variable's chunk cache at the HDF5 level (https://unidata.github.io/netcdf4-python/#netCDF4.Dataset.createVariable), and it would be useful to expose this to cf.

Currently the netCDF4 default value of 16777216 bytes (i.e. 16 MiB) is used.

Changing this has no effect on the new netCDF-4 file on disk, but may be used to prevent the available memory from filling up when a very large number of netCDF-4 variables are being created. For instance, if 1024 netCDF-4 variables are being created, then by default 17179869184 bytes (i.e. 16 GiB) of memory will be needed for their chunk caches, and if this is too much then the chunk cache should be reduced.

This will be largely implemented in cfdm==1.12.2.0 by NCAS-CMS/cfdm#329

Metadata

Metadata

Assignees

No one assigned

    Labels

    dataset writeRelating to writing datasetsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions