Thanks for the quick feedback! I think I addressed all of your comments in my latest commit.

I have to mark this as WIP because I found that there are alternative implementations (data formats) and I'm not sure which one to choose:

0. Before I posted this PR here for review, I had a version that was based on pandas' HDF5 io capabilities (be78053). However, I found that this produces files that are way to large. Further, it would use pickle internally to serialize strings, even if we have a fixed-length string variable like "basin". And pickle is really bad for backwards compatibility.

1. My initial proposal (when posting this PR) would encode each track separately as a NetCDF3 file and then store its (compressed) byte representation in the hdf5 file. This is not very elegant since it is not really using the hdf5 structure.

2. An alternative file format is in bf878c8 which produces NetCDF4-compliant HDF5-files that can also be inspected with NetCDF-tools like ncdump. However, the resulting hdf5 file is much (factor 4 or 5!) larger, even with zlib compression enabled. That's because it has huge amounts of metadata and metadata is not compressed. In the first format, the NetCDF headers (metadata) are compressed as well.

When gzipping the files in the format (2), the size problem is more than solved. However, it's not possible to read and write directly to/from gzipped hdf5 files when using xarray. We would first need to write the uncompressed data to disk and then gzip it. Similarly, when reading the file, we would need to uncompress the data on disk (which requires a lot of space) and then read it from there.

@chahank I'm sorry, but I have to ask again for your review. I didn't change the API or the tests, but I changed the file format and the implementation. The produced HDF5-file is now completely NetCDF4-compliant and can be read with external NetCDF-tools like ncdump. This is much more elegant for most purposes, but requires factor 4-5 more space on disk than with the data format you reviewed at first. For example, storing the appx. 3000 tracks from the global IBTrACS data set in 1980-2019 requires 87 MB on disk [*]. This doesn't sound like a lot, but when generating 100 synthetic random walk tracks for each event, you will end up with almost 9 GB of data.

However, users that have problems with disk space can still manually gzip their files which will usually save 90% of the disk space.

[*] For comparison, when storing the 3000 tracks in separate NetCDF files using the TCTracks.write_netcdf functionality, the total disk space required is 105 MB.

Thanks @tovogt . I went through the code, and made a few remarks.

I think having a method that makes use of the files easier and compatible at the cost of memory is the right choice. And, as you said, for user where it is critical they may use extra compressing methods.

Looks good to go to me, thanks for the updates!