Chunking fail for String variable with new netcdf-c package library (4.9.0) #1420
Description
Versions impacted by the bug
netcdf-java : 5.7.0
netcdf-c : 4.9.0 (packed with libnetcdf19)
What went wrong?
- Problem Statement: When the NetCDF-C native library version 4.9.0 or later is installed, the NetCDF-Java library exhibits unexpected failures during writing Netcdf4 file involving chunking. This issue does not occur with version 4.7.4.
- Expected Behavior: The tests should pass regardless of the NetCDF-C library version.
- Actual Behavior: The tests fail when using NetCDF-C version 4.9.0 or later.
netcdf4 file writing fail writing variable
String qualityFlags(dateEnd=1051776);
:flag_meanings = "missing_value Checked Estimated_from_bucket,valid_over_the_whole_period";
:standard_name = "quality_flag";
:flag_values = "11 9 12";
at
This issue is raised for String variable and is not thrown for other Double variable from the same file. This could be linked to Unidata/netcdf4-python#1205 but from this discussion this issue is raised when trying to use compression on variable length String variable which seems not to be my case.
Relevant stack trace
java.io.IOException: NetCDF: Filter error: bad id or parameters or duplicate filter
at ucar.nc2.jni.netcdf.Nc4Iosp.createVariable(Nc4Iosp.java:2546)
at ucar.nc2.jni.netcdf.Nc4Iosp.createGroup(Nc4Iosp.java:2458)
at ucar.nc2.jni.netcdf.Nc4Iosp.createGroup(Nc4Iosp.java:2469)
at ucar.nc2.jni.netcdf.Nc4Iosp.createGroup(Nc4Iosp.java:2469)
at ucar.nc2.jni.netcdf.Nc4Iosp.createGroup(Nc4Iosp.java:2469)
at ucar.nc2.jni.netcdf.Nc4Iosp.create(Nc4Iosp.java:2370)
at ucar.nc2.write.NetcdfFormatWriter.<init>(NetcdfFormatWriter.java:337)
at ucar.nc2.write.NetcdfFormatWriter.<init>(NetcdfFormatWriter.java:42)
at ucar.nc2.write.NetcdfFormatWriter$Builder.build(NetcdfFormatWriter.java:267)
at fr.theialand.insitu.data.netcdf.creation.service.NetcdfBuilderService.writeNetcdfFromParquetFileInput(NetcdfBuilderService.java:210)
at fr.theialand.insitu.data.netcdf.creation.service.NetcdfBuilderServiceTest.writeNetcdfFromParquetFileInput(NetcdfBuilderServiceTest.java:112)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)Relevant log messages
8922 [main] INFO ucar.nc2.jni.netcdf.Nc4wrapper - trace: nc_def_var ret=0 args=(65539,qualityFlags,12,1,[I@217b0952,int@0x7f68c9d22be0=0x6 (6))
8922 [main] INFO ucar.nc2.jni.netcdf.Nc4wrapper - trace: nc_def_var_chunking ret=0 args=(65539,6,0,[Lucar.nc2.jni.netcdf.SizeT;@3fd9e01c)
8922 [main] INFO ucar.nc2.jni.netcdf.Nc4wrapper - trace: nc_def_var_deflate ret=-132 args=(65539,6,1,1,5)
8922 [main] INFO ucar.nc2.jni.netcdf.Nc4wrapper - trace: nc_strerror ret=NetCDF: Filter error: bad id or parameters or duplicate filter args=(-132)
8924 [main] INFO ucar.nc2.jni.netcdf.Nc4wrapper - trace: nc_close ret=0 args=(65536)
To reproduce my issue : https://github.com/charlycou/netcdf4-java-chunking-issue