Reading of UM: reference to non-existent axis on single-level fields #641
Description
As noticed during the testing of the tutorial code during the release process (which I am postponing until we can determine the appropriate fix here, since I don't want to release without being sure any possible issue is patched!) via the running of tutorial.py, there is an issue with reading UM manifesting as the traceback shown below which that script runs into.
With a little investigation, it seems that the logic currently assumes there can be a z-axis in any case when setting the data and the data axes using create_data, but for single-level fields (nz=1) there might not be a z axis, as far as I can understand.
From a git blame and some reverting to the old code, this issue seems to originate from the PR #630 where the _axis[self.z_axis] call has been moved out from the if nz > 1: statement (apologies, I did not pick up on this aspect during the code review). To back this up, when I move the relevant line like so, essentially reverting partially back to the code before the nearby changes from that PR:
diff --git a/cf/read_write/um/umread.py b/cf/read_write/um/umread.py
index 04ef52d8b..62862b6ca 100644
--- a/cf/read_write/um/umread.py
+++ b/cf/read_write/um/umread.py
@@ -1970,8 +1970,9 @@ class UMField:
# ----------------------------------------------------
# 1-d partition matrix
# ----------------------------------------------------
- z_axis = _axis[self.z_axis]
if nz > 1:
+ z_axis = _axis[self.z_axis]
pmaxes = [z_axis]
data_shape = (nz, LBROW, LBNPT)
else:
@@ -1983,7 +1984,7 @@ class UMField:
indices = [(i, rec) for i, rec in enumerate(recs)]
- if z_axis in self.down_axes:
+ if nz > 1 and z_axis in self.down_axes:
indices = self._reorder_z_axis(indices, z_axis, pmaxes)
for i, rec in indices:the tutorial no longer hits than issue and runs (hitting a separate issue later, but at least getting started and running for a significant part of the script unlike without the diff here!). So a change similar to this, with the justification I provide here which seems logical but could do with a sanity check, sounds like a sensible change to fix what seems to be a genuine issue, rather than some facet of the script (which blame tells me hasn't changed in 4 years apart from some linting).
@davidhassell I will wait to complete the release until I can get your second opinion on whether this is the issue at hand and if my idea to fix it is along the right lines. Thanks.
Failure of tutorial.py on present main branch
$ ./test_tutorial_code
+ echo PYTHONPATH=/home/sadie/cf-python:
PYTHONPATH=/home/sadie/cf-python:
+ d=/home/sadie/cf-python
+ cd docs/source
+ pwd
/home/sadie/cf-python/docs/source
+ python extract_tutorial_code.py tutorial.rst field_analysis.rst
+ ls -l tutorial.py field_analysis.py
-rw-rw-r-- 1 sadie sadie 7132 Apr 26 22:24 field_analysis.py
-rw-rw-r-- 1 sadie sadie 28455 Apr 26 22:24 tutorial.py
+ ./reset_test_tutorial
+ sample_files=sample_files
+ test_tutorial=test_tutorial
+ [[ -d test_tutorial ]]
+ rm test_tutorial/air_temperature.nc test_tutorial/cf_tutorial_files.zip test_tutorial/contiguous.nc test_tutorial/external.nc test_tutorial/file2.nc test_tutorial/file.nc test_tutorial/gathered.nc test_tutorial/geometry.nc test_tutorial/parent.nc test_tutorial/precipitation_flux.nc test_tutorial/timeseries.nc test_tutorial/umfile.pp test_tutorial/vertical.nc test_tutorial/wind_components.nc
+ cp sample_files/air_temperature.nc sample_files/cf_tutorial_files.zip sample_files/contiguous.nc sample_files/external.nc sample_files/file2.nc sample_files/file.nc sample_files/gathered.nc sample_files/geometry.nc sample_files/parent.nc sample_files/precipitation_flux.nc sample_files/timeseries.nc sample_files/umfile.pp sample_files/vertical.nc sample_files/wind_components.nc test_tutorial
+ cd test_tutorial
+ zip cf_tutorial_files.zip air_temperature.nc contiguous.nc external.nc file2.nc file.nc gathered.nc geometry.nc parent.nc precipitation_flux.nc timeseries.nc vertical.nc wind_components.nc umfile.pp
updating: air_temperature.nc (deflated 42%)
updating: contiguous.nc (deflated 57%)
updating: external.nc (deflated 68%)
updating: file2.nc (deflated 40%)
updating: file.nc (deflated 63%)
updating: gathered.nc (deflated 48%)
updating: geometry.nc (deflated 69%)
updating: parent.nc (deflated 67%)
updating: precipitation_flux.nc (deflated 66%)
updating: timeseries.nc (deflated 75%)
updating: vertical.nc (deflated 60%)
updating: wind_components.nc (deflated 84%)
updating: umfile.pp (deflated 45%)
+ set +x
+ cd test_tutorial
+ pwd
/home/sadie/cf-python/docs/source/test_tutorial
+ python ../tutorial.py
WARNING: Can't determine format of file /home/sadie/cf-python/docs/source/test_tutorial/cf_tutorial_files.zip
Traceback (most recent call last):
File "../tutorial.py", line 14, in <module>
y = cf.read('$PWD', ignore_read_error=True)
File "/home/sadie/cfdm/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
File "/home/sadie/cf-python/cf/read_write/read.py", line 932, in read
file_contents = _read_a_file(
File "/home/sadie/cfdm/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
File "/home/sadie/cf-python/cf/read_write/read.py", line 1193, in _read_a_file
out = UM.read(
File "/home/sadie/cfdm/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
File "/home/sadie/cf-python/cf/read_write/um/umread.py", line 3467, in read
um = [
File "/home/sadie/cf-python/cf/read_write/um/umread.py", line 3468, in <listcomp>
UMField(
File "/home/sadie/cf-python/cf/read_write/um/umread.py", line 1040, in __init__
data = self.create_data()
File "/home/sadie/cf-python/cf/decorators.py", line 161, in verbose_override_wrapper
return method_using_verbose_attr(self, *args, **kwargs)
File "/home/sadie/cf-python/cf/read_write/um/umread.py", line 1973, in create_data
z_axis = _axis[self.z_axis]
KeyError: 'z'
+ rc=1
+ [[ 1 != 0 ]]
+ echo 'Runtime failure in tutorial.py'
Runtime failure in tutorial.py
+ exit 1Environment
>>> cf.environment()
Platform: Linux-4.15.0-54-generic-x86_64-with-glibc2.10
HDF5 library: 1.14.0
netcdf library: 4.9.2
udunits2 library: /home/sadie/anaconda3/envs/cf-env/lib/libudunits2.so.0
esmpy/ESMF: 8.4.1 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/esmpy/__init__.py
Python: 3.8.16 /home/sadie/anaconda3/envs/cf-env/bin/python
dask: 2023.1.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/__init__.py
netCDF4: 1.6.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/netCDF4/__init__.py
psutil: 5.9.4 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/psutil/__init__.py
packaging: 23.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/packaging/__init__.py
numpy: 1.24.2 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/__init__.py
scipy: 1.10.1 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/scipy/__init__.py
matplotlib: 3.4.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/matplotlib/__init__.py
cftime: 1.6.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cftime/__init__.py
cfunits: 3.3.5 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.1.18 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfplot/__init__.py
cfdm: 1.10.1.0 /home/sadie/cfdm/cfdm/__init__.py
cf: 3.15.0 /home/sadie/cf-python/cf/__init__.py