Config Templating by jammont · Pull Request #650 · isofit/isofit

jammont · 2025-03-12T21:35:51Z

Added new functionality to ini.py to convert to/from template configuration files which will make it easier to move a config from one system to another.

toTemplate converts an existing config file to a template version
- This must be done by the system that created the config to start with as it uses the ini to replace values in strings
- Values found in the ini are replaced with strings {env.[path]}, eg. {env.data}, {env.examples}
- By default, only converts paths from the ini, not keys. This is to ensure things like sRTMnet version remains consistent.
- template_construction.py auto does this now for presolve and full, creates the same config but with .tmpl
fromTemplate converts a template config back using a different ini. The idea here is:
- A user runs apply_oe.py
- full.json and full.json.tmpl config files are generated
- User hits an error, seeks help, uploads their entire working_directory (ie. output dir)
- Devs use options to convert:
  - Python:
```
>>> env.fromTemplate("/path/to/working_directory/config/full.json.tmpl", working_directory="/path/to/working_directory")
```
  - CLI:
```
$ isofit dev fromtmpl /path/to/working_directory/config/full.json.tmpl -k working_directory /path/to/working_directory
```
  - Note the additional keyword working_directory, this is not found in the ini but is an additional replaceable item in the template
- Devs execute config, should be fully working with whatever environment / ini is loaded
If a given config is detected to be within a valid ISOFIT output directory then the working_directory will be auto-set. If not, the user will have to set it (such as the examples)
isofit.py will attempt to call the fromTemplate function if it detects an input config is a template version, and then use the converted return

Additionally, added the above capabilities to our CLI:

isofit dev

$ isofit dev
Usage: isofit dev [OPTIONS] COMMAND [ARGS]...

  Extra commands for developers

Options:
  --help  Show this message and exit.

Commands:
  fromtmpl  Convert a template to a config
  totmpl    Convert a config to a template

isofit dev totmpl

$ isofit dev totmpl
Usage: isofit dev totmpl [OPTIONS] DATA

  Recursively converts string values in a dict to be template values which can
  be converted back using Ini.fromTemplate(). Template values are in the form
  of "{env.[value]}".

  Parameters
  ----------
  data : str | dict
      The dictionary to walk over and update values. If string, checks if this
      exists as a file and loads that in as the data dict
  replace : "dirs" | "keys" | None, default="dirs"
      Defines what kind of values from the ini to replace in strings:
      - "dirs" only replace directory paths
      - "keys" only replace key strings
      - None replaces both
      Recommended to only use "dirs" to remain consistent. "keys" can have
      unintended consequences and may replace more than it should
  save : bool, default=True
      If the data was a file and this is enabled, saves the converted data dict
      to another file. The new file will simply append ".tmpl" to its name
  **kwargs : dict
      Additional strings to replace. The values are replaced in a string with the
      key of the kwarg. For example:
      >>> kwargs = {"xyz": "abc"}
      >>> data["some_key"] = "replace abc here"
      will be replaced as:
      >>> data["some_key"] = "replace {xyz} here"
      This is to be used with Ini.fromTemplate to replace values that are not
      found in the ini object

  Returns
  -------
  data : dict
      In-place replaced string values with template values

Options:
  -r, --replace [dirs|keys|None]
  -ns, --no-save                  Disables saving to file, will print to
                                  terminal
  -k, --kwargs TEXT...            Example: -k working_directory
                                  /path/to/output
  --help                          Show this message and exit.

isofit dev fromtmpl

$ isofit dev fromtmpl
Usage: isofit dev fromtmpl [OPTIONS] DATA

  Recursively replaces the template values in found in string values with the
  real value from the ini. Template values are in the form of "{env.[value]}".
  This is an in-place operation.

  Parameters
  ----------
  data : str | dict
      The dictionary to walk over and update values. If string, checks if this
      exists as a file and loads that in as the data dict
  save : bool, default=True
      If the data was a file and this is enabled, saves the converted data dict
      to another file. If the input file ends with ".tmpl" then it will simply be
      cut. If it doesn't or already exists, then the output filename will be the
      input filename prepended with `prepend` value.
  prepend : str, default=None
      Prepend a string to the output filename. If not set and the input filename
      doesn't end with ".tmpl", then this is auto-set to "replaced"
  **kwargs : dict
      Additional strings to replace. The values are replaced in a string with the
      key of the kwarg. For example:
      >>> kwargs = {"xyz": "abc"}
      >>> data["some_key"] = "replace {xyz} here"
      will be replaced as:
      >>> data["some_key"] = "replace abc here"
      This is to be used with Ini.toTemplate to replace values that are not found
      in the ini object

  Returns
  -------
  data : dict
      In-place replaced template values with actual from a loaded ini

Options:
  -ns, --no-save        Disables saving to file, will print to terminal
  -p, --prepend TEXT
  -k, --kwargs TEXT...  Example: -k working_directory /path/to/output
  --help                Show this message and exit.

Example

Trimmed the configs some so they're not so long

Working config generated by `template_construction.py`

{
    "forward_model": {
        "instrument": {
            "integrations": 200,
            "parametric_noise_file": "/store/jamesmo/isofit/extras/data/emit_noise.txt",
            "unknowns": {
                "channelized_radiometric_uncertainty_file": "/local/jemit/data/channelized_uncertainty.txt",
                "uncorrelated_radiometric_uncertainty": 0.01
            },
            "wavelength_file": "/local/jemit/data/wavelengths.txt"
        },
        "model_discrepancy_file": "/local/jemit/data/model_discrepancy.mat",
        "radiative_transfer": {
            "radiative_transfer_engines": {
                "vswir": {
                    "aerosol_model_file": "/store/jamesmo/isofit/extras/data/aerosol_model.txt",
                    "aerosol_template_file": "/store/jamesmo/isofit/extras/data/aerosol_template.json",
                    "earth_sun_distance_file": "/store/jamesmo/isofit/extras/data/earth_sun_distance.txt",
                    "emulator_aux_file": "/store/jamesmo/isofit/extras/srtmnet/sRTMnet_v120_aux.npz",
                    "emulator_file": "/store/jamesmo/isofit/extras/srtmnet/sRTMnet_v120.h5",
                    "engine_base_dir": "/store/jamesmo/isofit/extras/sixs",
                    "engine_name": "sRTMnet",
                    "glint_model": false,
                    "interpolator_base_path": "/local/jemit/lut_full/sRTMnet_v120_vi",
                    "irradiance_file": "/store/jamesmo/isofit/extras/examples/20151026_SantaMonica/data/prism_optimized_irr.dat",
                    "lut_names": {
                        "AOT550": null,
                        "H2OSTR": null,
                        "observer_zenith": null,
                        "relative_azimuth": null,
                        "surface_elevation_km": null
                    },
                    "lut_path": "/local/jemit/lut_full/lut.nc",
                    "multipart_transmittance": false,
                    "sim_path": "/local/jemit/lut_full",
                    "statevector_names": [
                        "H2OSTR",
                        "surface_elevation_km",
                        "AOT550"
                    ],
                    "template_file": "/local/jemit/config/emit20220820t131606_modtran_tpl.json"
                }
            },
        },
        "surface": {
            "select_on_init": true,
            "surface_category": "multicomponent_surface",
            "surface_file": "/local/jemit/data/surface.mat"
        }
    },
    "input": {
        "loc_file": "/local/jemit/input/emit20220820t131606_subs_loc",
        "measured_radiance_file": "/local/jemit/input/emit20220820t131606_subs_rdn",
        "obs_file": "/local/jemit/input/emit20220820t131606_subs_obs"
    },
    "output": {
        "atmospheric_coefficients_file": "/local/jemit/output/emit20220820t131606_subs_atm",
        "estimated_reflectance_file": "/local/jemit/output/emit20220820t131606_subs_rfl",
        "estimated_state_file": "/local/jemit/output/emit20220820t131606_subs_state",
        "posterior_uncertainty_file": "/local/jemit/output/emit20220820t131606_subs_uncert"
    }
}

Template config generated by `template_construction.py`

{
    "forward_model": {
        "instrument": {
            "integrations": 200,
            "parametric_noise_file": "{env.data}/emit_noise.txt",
            "unknowns": {
                "channelized_radiometric_uncertainty_file": "{working_directory}/data/channelized_uncertainty.txt",
                "uncorrelated_radiometric_uncertainty": 0.01
            },
            "wavelength_file": "{working_directory}/data/wavelengths.txt"
        },
        "model_discrepancy_file": "{working_directory}/data/model_discrepancy.mat",
        "radiative_transfer": {
            "radiative_transfer_engines": {
                "vswir": {
                    "aerosol_model_file": "{env.data}/aerosol_model.txt",
                    "aerosol_template_file": "{env.data}/aerosol_template.json",
                    "earth_sun_distance_file": "{env.data}/earth_sun_distance.txt",
                    "emulator_aux_file": "{env.srtmnet}/sRTMnet_v120_aux.npz",
                    "emulator_file": "{env.srtmnet}/sRTMnet_v120.h5",
                    "engine_base_dir": "{env.sixs}",
                    "engine_name": "sRTMnet",
                    "glint_model": false,
                    "interpolator_base_path": "{working_directory}/lut_full/sRTMnet_v120_vi",
                    "irradiance_file": "{env.examples}/20151026_SantaMonica/data/prism_optimized_irr.dat",
                    "lut_names": {
                        "AOT550": null,
                        "H2OSTR": null,
                        "observer_zenith": null,
                        "relative_azimuth": null,
                        "surface_elevation_km": null
                    },
                    "lut_path": "{working_directory}/lut_full/lut.nc",
                    "multipart_transmittance": false,
                    "sim_path": "{working_directory}/lut_full",
                    "statevector_names": [
                        "H2OSTR",
                        "surface_elevation_km",
                        "AOT550"
                    ],
                    "template_file": "{working_directory}/config/emit20220820t131606_modtran_tpl.json"
                }
            },
        },
        "surface": {
            "select_on_init": true,
            "surface_category": "multicomponent_surface",
            "surface_file": "{working_directory}/data/surface.mat"
        }
    },
    "input": {
        "loc_file": "{working_directory}/input/emit20220820t131606_subs_loc",
        "measured_radiance_file": "{working_directory}/input/emit20220820t131606_subs_rdn",
        "obs_file": "{working_directory}/input/emit20220820t131606_subs_obs"
    },
    "output": {
        "atmospheric_coefficients_file": "{working_directory}/output/emit20220820t131606_subs_atm",
        "estimated_reflectance_file": "{working_directory}/output/emit20220820t131606_subs_rfl",
        "estimated_state_file": "{working_directory}/output/emit20220820t131606_subs_state",
        "posterior_uncertainty_file": "{working_directory}/output/emit20220820t131606_subs_uncert"
    }
}

Template converted to my local Mac

Below commands generate the same thing

>>> env.fromTemplate("emit20220820t131606_isofit.json.tmpl", working_directory="/example/directory/replacement")

$ isofit dev fromtmpl emit20220820t131606_isofit.json.tmpl -k working_directory /example/directory/replacement

{
    "forward_model": {
        "instrument": {
            "integrations": 200,
            "parametric_noise_file": "/Users/jamesmo/projects/isofit/extras/data/emit_noise.txt",
            "unknowns": {
                "channelized_radiometric_uncertainty_file": "/example/directory/replacement/data/channelized_uncertainty.txt",
                "uncorrelated_radiometric_uncertainty": 0.01
            },
            "wavelength_file": "/example/directory/replacement/data/wavelengths.txt"
        },
        "model_discrepancy_file": "/example/directory/replacement/data/model_discrepancy.mat",
        "radiative_transfer": {
            "radiative_transfer_engines": {
                "vswir": {
                    "aerosol_model_file": "/Users/jamesmo/projects/isofit/extras/data/aerosol_model.txt",
                    "aerosol_template_file": "/Users/jamesmo/projects/isofit/extras/data/aerosol_template.json",
                    "earth_sun_distance_file": "/Users/jamesmo/projects/isofit/extras/data/earth_sun_distance.txt",
                    "emulator_aux_file": "/Users/jamesmo/projects/isofit/extras/srtmnet/sRTMnet_v120_aux.npz",
                    "emulator_file": "/Users/jamesmo/projects/isofit/extras/srtmnet/sRTMnet_v120.h5",
                    "engine_base_dir": "/Users/jamesmo/projects/isofit/extras/sixs",
                    "engine_name": "sRTMnet",
                    "glint_model": false,
                    "interpolator_base_path": "/example/directory/replacement/lut_full/sRTMnet_v120_vi",
                    "irradiance_file": "/Users/jamesmo/projects/isofit/extras/examples/20151026_SantaMonica/data/prism_optimized_irr.dat",
                    "lut_names": {
                        "AOT550": null,
                        "H2OSTR": null,
                        "observer_zenith": null,
                        "relative_azimuth": null,
                        "surface_elevation_km": null
                    },
                    "lut_path": "/example/directory/replacement/lut_full/lut.nc",
                    "multipart_transmittance": false,
                    "sim_path": "/example/directory/replacement/lut_full",
                    "statevector_names": [
                        "H2OSTR",
                        "surface_elevation_km",
                        "AOT550"
                    ],
                    "template_file": "/example/directory/replacement/config/emit20220820t131606_modtran_tpl.json"
                }
            },
        },
        "surface": {
            "select_on_init": true,
            "surface_category": "multicomponent_surface",
            "surface_file": "/example/directory/replacement/data/surface.mat"
        }
    },
    "input": {
        "loc_file": "/example/directory/replacement/input/emit20220820t131606_subs_loc",
        "measured_radiance_file": "/example/directory/replacement/input/emit20220820t131606_subs_rdn",
        "obs_file": "/example/directory/replacement/input/emit20220820t131606_subs_obs"
    },
    "output": {
        "atmospheric_coefficients_file": "/example/directory/replacement/output/emit20220820t131606_subs_atm",
        "estimated_reflectance_file": "/example/directory/replacement/output/emit20220820t131606_subs_rfl",
        "estimated_state_file": "/example/directory/replacement/output/emit20220820t131606_subs_state",
        "posterior_uncertainty_file": "/example/directory/replacement/output/emit20220820t131606_subs_uncert"
    }
}

…d the ability to replace them

…vert an existing config

evan-greenbrg · 2025-03-13T15:35:51Z

        )

+    # Create a template version of the config
+    env.toTemplate(


This is great @jammont! It's nice that it's non-invasive and only adds useful functionality. The only comment/question I had: is it the case that the template files are always generated?

Would it be useful to have a flag to turn off/on this functionality? (who doesn't love more function arguments)

This is automatic and cannot be disabled at this time. I figure the cost of the template files is insignificant and can be safely ignored. Having them auto-generated ensures that users that are unsure cannot mess-up and fail to provide them to us if they need assistance.

If we don't like them being generated side-by-side the source configs, we could create a new directory under the working_directory like debug or something to dump them to

…eplace a template config automatically

evan-greenbrg · 2025-04-16T21:32:13Z

@jammont I tried again to get this to work with no luck. Does this only work with configs generated via apply_oe with this branch?

I was attempting to use it to generate/rebuild template configs for a couple tests I'm trying to put together and the CLI commands were just returning the same input file.

…prints

evan-greenbrg · 2025-04-16T23:21:04Z

@jammont and I went back and forth and now I've gotten it to work locally, both totmpl and fromtmpl. I'm okay with merging this in. I think it will be really nice to have.

Two thoughts:

James' description at the top is a really nice description of the functionality. Is that information located anywhere else?
I can raise an issue on this, but it would be nice to merge this template structure with the example template we use for workflow testing. It would make configuring the template files for workflows a breeze.

James Montgomery added 6 commits March 12, 2025 13:05

Implemented functions to generate configs using env template paths an…

6366e2e

…d the ability to replace them

Renamed ini.replace to ini.fromTemplate; create ini.toTemplate to con…

33c5788

…vert an existing config

Removed template conversion in template_construction

9eafda5

Forgot self

4284c43

Added extra replacement functionality to the templating functions

edacb28

Pass kwargs recursively

b4da16e

jammont added the enhancement New feature or request label Mar 12, 2025

Added ini templating functions to cli

4a8dd9b

jammont requested review from evan-greenbrg, pgbrodrick and unbohn March 12, 2025 23:29

evan-greenbrg reviewed Mar 13, 2025

View reviewed changes

James Montgomery added 2 commits March 13, 2025 12:53

Added autodiscovery of working_directory; isofit.py will attempt to r…

175c088

…eplace a template config automatically

Fixed missing import

e50b643

Fixed totmpl cli using wrong func; added more logging info; replaced …

fee499a

…prints

evan-greenbrg self-requested a review April 16, 2025 23:08

evan-greenbrg approved these changes Apr 16, 2025

View reviewed changes

pgbrodrick merged commit 89848ce into isofit:dev May 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Config Templating#650

Config Templating#650
pgbrodrick merged 10 commits into
isofit:devfrom
jammont:debug/configs

jammont commented Mar 12, 2025 •

edited

Loading

Uh oh!

evan-greenbrg Mar 13, 2025

Uh oh!

jammont Mar 13, 2025

Uh oh!

evan-greenbrg commented Apr 16, 2025

Uh oh!

evan-greenbrg commented Apr 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jammont commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example

Uh oh!

evan-greenbrg Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jammont Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

evan-greenbrg commented Apr 16, 2025

Uh oh!

evan-greenbrg commented Apr 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jammont commented Mar 12, 2025 •

edited

Loading