new EOF capability#830
Conversation
|
Here is an example of the use of two EOFs to remove CO2 and H2O-related residuals in the new EMIT candidate v2 reflectance configuration |
|
This looks great! Thanks for adding this nice feature, @davidraythompson. My two comments are only minor and the PR is basically ready to go in my view. Also, I took the liberty and ran black on your changes. |
| @@ -363,21 +368,36 @@ def dmeas_dinstrumentb(self, x_instrument, wl_hi, rdn_hi): | |||
| def sample(self, x_instrument, wl_hi, rdn_hi): | |||
There was a problem hiding this comment.
Apologies because I'm still trying to get my head around what the EOFs physically correspond to. Is it correct to think about them as a composite shift in radiance across n-number of EOFs? I'm sure the magic is in how the EOF curves are fit.
Either way, we are currently applying the same scaled EOF offset for both the resampling of the prediction of the measurement as well as the resampling of the derivative of the measurement w.r.t. to statevector elements
Shift on the predicted measurement makes sense to me
pred_meas = self.instrument.sample(x_instrument, self.RT.wl, rdn)
Shift on the derivatives I'm still fuzzy on:
e.g. dmeas_dRT = self.instrument.sample(x_instrument, self.RT.wl, drdn_dRT.T).T
I haven't worked the math out at all, but is this intended? The shared spectra speak for themselves so I must be missing something.
There was a problem hiding this comment.
Per our discussion in the in-person meeting: you're right, this should apply to radiances only. Your proposed fix works great. Nice catch!
EOF edits - thanks to Phil Brodrick and Evan Greenberg.
Adds a reference to EMIT's EOF file contained in the isofit-data repository
|
Remember to update isofit-data (isofit/isofit-data#10) to test this out! |
|
@davidraythompson @pgbrodrick Except for my comments above, all tests were successful and resulting EMIT spectra look great. Once those minor issues are resolved, we should be ready to merge this in. |
|
@pgbrodrick After testing the 6c emulator, I had to push a small fix to the new general config build function. I'll merge once all checks have passed. |
|
All issues are resolved, checks have passed, ready for merge. Thanks, @davidraythompson! |
Added an Empirical Orthogonal Function capability to our retrievals, as described in O'Dell et al (https://amt.copernicus.org/articles/11/6539/2018/). This seems to do a good job of removing variable CO2-related residuals at 2000-2050 nm in EMIT data, making it a candidate for inclusion in the next version of those reflectance products.
The basic approach is to (1) characterize the shape of the atmospheric residuals in advance, and then (2) at runtime, add this shape to the modeled radiance with a scaling coefficient that is retrieved dynamically. This helps the retrieval fit atmospheric features easily without having to bend the reflectance.