Something we could consider for this PR or subsequent ones, is that we could increase the complexity and store x0 as well as bg_rfl. I coded this out and realized there was too many changes at once and so my preference I think would be to only have the bg_rfl first in this PR to isolate any errors/bugs. This was suggested by Evan this week too in that we could essentially remove the inverse_simple() call wehn running OE. The x0 potentially could be stored in geom?

@unbohn Comparing similar pixels in emit20250327T212148_rfl over Lake Mary, CA. surface_20240103.json. All were done with similar configs really just trying to sub out the emulator and the background pixel assumptions. Skyview was assumed to be 1.0 for all here.

--

The resulting bg_rfl spectra for this same location ^^

Super interesting @brentwilder! I was curious why the analytical line w/ bkg solve didn't better match with the OE. I actually think as it's currently written, the two forward models (OE vs AOE) are not equivalent.

The linearization we do in the analytical line in its present form assumes that rho_dif_dir = rho_dif_dif and rho_dir_dir = rho_dir_dif. Is it true that this is not the case in the OE forward model where these two are equal to the geom.bg_rfl?

If this is the case, we just need to back out the non-simplified form of the linearized forward model and hook up geom.bg_rfl appropriately.

Full forward model:
$L=L_0+L_{11}\rho_{11}+L_{21}\rho_{21}+L_{12}\rho_{12}+L_{22}\rho_{22}+\frac{L_TS\rho_{22}^2}{1-S\rho_{22}}$

Assumptions:
$\rho_{12}=\rho_{22}=\rho_{Bg}$
$\rho_{11}=\rho_{21}=\rho$

Simplifies to
$L=L_0+(L_{11}+L_{21})\rho+(L_{12}+L_{22})\rho_{Bg}+\frac{L_TS\rho_{Bg}^2}{1-S\rho_{Bg}}$

Assume:
$L_{Bg} = (L_{12}+L_{22})\rho_{Bg}+\frac{L_TS\rho_{Bg}^2}{1-S\rho_{Bg}}$

AOE solves for the following linear fn w.r.t state-vector, $\rho$:
$L-L_0-L_{Bg}=(L_{11}+L_{21})\rho$

Is this right?

Edit: with the glint model:

Assuming:
$\rho_{11}=\rho + \alpha_1$
$\rho_{21}=\rho + \alpha_2$

AOE solves for the following linear fn w.r.t state-vector, $\rho, \alpha_1,\alpha_2$:
$L-L_0-L_{Bg}=(L_{11}+L_{21})\rho+L_{11}\alpha_1+L_{21}\alpha_2$

May actually help stability...

The tricky thing is how this can be "architectured" to be flexible w.r.t running w/ background and w/o background.

Thanks for these updates, guys! @brentwilder I think your plot tells exactly the story that the current AOE implementation sets the target reflectance equal to the background, so that we can't get a match to the per pixel solve that includes the background. @evan-greenbrg I skimmed your equations and they look right to me. To your last point: In my opinion, we should always run with background, as the diffuse reflectance will never originate from the target pixel alone.

@unbohn @evan-greenbrg On your last point Niklas I'd agree that being computationally cheap, you would always want to have this mode on. But I can definitely attempt to write it out to be optional, and we can decide on it as a group.

In writing this I realized also that the full bg_rfl file needs to be shared to analytical line call , to facilitate these above equations. But I think the way Evan has mapped it out here is very nice and can be used to improve AOE ... Essentially even for the 1C case, instead of using the superpixel $$\rho$$ we could always rely on the geom.bg_rfl , here .

@brentwilder Yes, I fully agree! We've had some discussions about the usage of the background reflectance term in both OE and AOE a long while ago, and I think David came up with the idea of using the superpixel reflectance as background. Apparently, nobody thought about utilizing the heuristic solve back then.

6c analytical solution after Evan's suggestions. My interpretation of the 1c case was to assume $$\theta \approx L_{tot}$$ . And the new L_bg term handles the constant spherical albedo scattering in xk (Line 366 of inverse_simple).

Although this 1c case exceeds the 95th percentile of ASD measurements, I think the math is pretty close here.. Hard to tell if this could just be a limitation of using 1c based radiance term. Or perhaps my math is wrong, which is certainly possible.

To make this also useful for the per pixel and LUTSurface solutions, I expanded the meaning of use_superpixel a bit, such that its True to start ApplyOE so it can leverage speed up for presolve/background solve.

I'd assume we would never want to do the background solve without super pixel.

And then there is a check right before main config is built to set it back to False if AOE or Empirical Line is not turned on.

The main improvement from this latest commit is that it would allow @unbohn and I to run the Snow model with the advancements here in this PR.

URL: isofit/isofit-test-results#21
SHA: 59c0e72

Ready for vigorous testing now.