Skip to content

Conversation

@lbdreyer
Copy link
Member

@lbdreyer lbdreyer commented May 11, 2021

We recently had a user who wanted to load in data with grib data template number 42, i.e. CCSDS with compression, which we currently do not support.

The data decompression is all done in eccodes, so all we need to get this new template working is

  • ensure we have a build of eccodes which was built with AEC enabled (the conda-forge build of eccodes already has this enabled),
  • add 42 to the list of supported template numbers
    then when iris-grib loads in the data from the grib file, it goes via eccodes, which handles the decompresssion, and so returns a numpy array as normal.

I have kept the code change to iris-grib very lightweight (so I didn't check that the eccodes version running was built with AEC enabled) as all the handling of the data is performed by eccodes so we should just hand that off to eccodes.
If you're interested. if you try to run my branch in an env without AEC, you get the following error:

ECCODES ERROR   :  grib_accessor_data_ccsds_packing: CCSDS support not enabled. Please rebuild with -DENABLE_AEC=ON (Adaptive Entropy Coding library)
Traceback (most recent call last):
  File "something.py", line 42, in <module>
    print(cube.data[0,0])
 ...
  File ".../lib/python3.7/site-packages/iris_grib/message.py", line 444, in _get_key_value
    res = gribapi.grib_get_array(self._message_id, key)
  File ".../lib/python3.7/site-packages/gribapi/gribapi.py", line 1958, in grib_get_array
    result = grib_get_double_array(msgid, key)
  File ".../lib/python3.7/site-packages/gribapi/gribapi.py", line 1170, in grib_get_double_array
    GRIB_CHECK(err)
  File ".../lib/python3.7/site-packages/gribapi/gribapi.py", line 225, in GRIB_CHECK
    errors.raise_grib_error(errid)
  File ".../lib/python3.7/site-packages/gribapi/errors.py", line 382, in raise_grib_error
    raise ERROR_MAP[errid](errid)
gribapi.errors.FunctionalityNotEnabledError: Functionality not enabled

which is informative enough. I have decided against catching this error in a try/except block and instead putting out our own error. That seems unnecessary.

As for tests I have followed a similarly lightweight approach. As all we're doing in iris-grib is checking the template number against a list of known supported templates, all I've done is test this. I haven't tested that what eccodes returns to us is correctly decompressed data as that should be tested by eccodes.

Copy link
Member

@pp-mo pp-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm basically good with this approach
( But I still have to test practically with the test-data + the env you provided locally. )

Meantime, this test needs the tiny fix I think.

…tion.py

Co-authored-by: Patrick Peglar <patrick.peglar@metoffice.gov.uk>
@pp-mo pp-mo merged commit bdb8c63 into SciTools:master May 12, 2021
@lbdreyer lbdreyer deleted the aec branch June 27, 2021 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants