Download and convert satellite data for use in ML pipelines

Satellite data is a valuable resource for training machine learning models. Forecasting renewable generation requires knowledge of the weather conditions, and those weather conditions can be inferred and enriched using satellite data.

EUMETSAT provide a range of satellite data products, which are easily available in NAT image format. In order to improve its accessibility for training models, this consumer processes downloaded data into the Zarr format.

Install using the container image:

$ docker pull ghcr.io/openclimatefix/satellite-consumer

$ docker run \
    -e <SATCONS_CONFIG_VARIABLE>=<your-value>
    -e EUMETSAT_CONSUMER_KEY=<your-key> \
    -e EUMETSAT_CONSUMER_SECRET=<your-secret> \
    -v $(pwd)/work:/work \
    ghcr.io/openclimatefix/satellite-consumer

This will download the latest available data for the rss satellite and store it in the /work directory. For a description of all the possible configuration options, see Documentation.

There are a number of configuration options exposed via environment variables. For the full list, see `cmd/application.conf.

Currently the consumer is built to the specific data requirements of Open Climate Fix. However, adding a new satellite in the from EUMETSAT shouldn't be too hard, provided it uses the same seviri_l1b_native format and sensor channels - just update the available satellites in config.py.

OCF recommends using uv for managing your virtual environments.

$ git clone git@github.com:openclimatefix/satellite-consumer.git
$ cd satellite-consumer

Initialize the repository with recommended settings for development via

$ make init

This project uses MyPy for static type checking and Ruff for linting. Installing the development dependencies makes them available in your virtual environment.

There is a makefile target to automatically lint, typecheck, and format the codebase:

$ make lint

Be sure to do this periodically while developing to catch any errors early and prevent headaches with the CI pipeline. It may seem like a hassle at first, but it prevents accidental creation of a whole suite of bugs.

There are some additional dependencies to be installed for running the tests, be sure to pass --extra=dev to the pip install -e . command when creating your virtualenv (uv sync includes the development dependencies by default, so uv users can ignore this!).

Run the unit tests with:

$ make test

On the directory structure:

• The official PyPA discussion on "source" and "flat" layouts.

• PR's are welcome! See the Organisation Profile for details on contributing
• Find out about our other projects in the here
• Check out the OCF blog for updates
• Follow OCF on LinkedIn

devsjc 💻

Jacob Prince-Bieker 💻

Peter Dudfield 🐛

Ramkumar R 💻

Part of the Open Climate Fix community.