Skip to content

Numba grid interpolator#885

Merged
evan-greenbrg merged 3 commits into
isofit:devfrom
pgbrodrick:numbagrid
Mar 12, 2026
Merged

Numba grid interpolator#885
evan-greenbrg merged 3 commits into
isofit:devfrom
pgbrodrick:numbagrid

Conversation

@pgbrodrick

@pgbrodrick pgbrodrick commented Mar 2, 2026

Copy link
Copy Markdown
Collaborator

On a speedtest as defined below, gives:

Orig: 0.277
New: 0.0456

A factor of 6ish. TBD what that means for full isofit runs. Results will vary depending on grid size. Also TBD if this should be default.

Numba is not bound in pyproj, we may need a minimum.

Speedtest (mostly borrowed from test):

import scipy

from isofit.core.common import (
    VectorInterpolator,
    combos,
    eps,
    expand_path,
    get_absorption,
    load_spectrum,
    load_wavelen,
    recursive_replace,
    spectral_response_function,
    svd_inv,
    svd_inv_sqrt,
)

grid_input = [[1, 5, 10], [2, 4, 6, 7], [50, 60, 80], [0.1, 0.5], [380, 420]]
data_input = np.random.random(
    (
        len(grid_input[0]),
        len(grid_input[1]),
        len(grid_input[2]),
        len(grid_input[3]),
        len(grid_input[4]),
        30,
    )
)

v_orig = VectorInterpolator(grid_input, data_input, version="mlg")
v_new = VectorInterpolator(grid_input, data_input, version="mlg_numba")

input_test = np.random.random((10000, len(grid_input)))
for _n in range(len(grid_input)):
    input_test[:, _n] = input_test[:, _n] * (
        np.max(grid_input[_n]) - np.min(grid_input[_n])
    ) + np.min(grid_input[_n])

res_orig = np.zeros((input_test.shape[0], data_input.shape[-1]))
res_new = np.zeros((input_test.shape[0], data_input.shape[-1]))

st = time.time()
for _n in range(res_orig.shape[0]):
    res_orig[_n, :] = v_orig(input_test[_n, :])
print(f'Orig: {time.time() - st}')
st = time.time()
for _n in range(res_orig.shape[0]):
    res_new[_n, :] = v_new(input_test[_n, :])
print(f'New: {time.time() - st}')

slope, intercept, rvalue, pvalue, stderr = scipy.stats.linregress(
    res_orig.flatten(), res_new.flatten()
)
assert rvalue**2 > 1 - 1e-6

@pgbrodrick

pgbrodrick commented Mar 2, 2026

Copy link
Copy Markdown
Collaborator Author

On large-scale apply_oe tests, for the presolve this seems to be about 2x faster and give solutions in the right range.

For the main solve, things become incredibly slow. Could be a dimension thing, could be an implementation bug, or could be that I'm not holding settings correctly in all situations.

Promising though!

@evan-greenbrg evan-greenbrg mentioned this pull request Mar 4, 2026
Closed
@pgbrodrick

pgbrodrick commented Mar 9, 2026

Copy link
Copy Markdown
Collaborator Author

With this updated version, the serial test (shown above), gives approximately the same result. A batched ray-based implementation of the test shows a 2x speedup (delta indicative of ray overhead)....but this version no longer massively fails when coupled with ray. Testing on full scenes to get a handle on impact.

@pgbrodrick

pgbrodrick commented Mar 9, 2026

Copy link
Copy Markdown
Collaborator Author

Confirmed results in AV5 test case:

Presolve:
2.5962 spectra/s/core -> 4.5048 spectra/s/core.
Speedup: 1.9x

Main Solution:
0.2974 spectra/s/core -> 0.7204 spectra/s/core
Speedup: 2.42x

Analytical Line:
23.7877 spectra/s/core -> 22.6507 spectra/s/core
Speedup: 0.95x

Total Time:
~88 minutes -> ~56 minutes
Speedup: 1.57x

@pgbrodrick pgbrodrick marked this pull request as ready for review March 9, 2026 04:21
unbohn
unbohn reviewed Mar 9, 2026
View reviewed changes
Comment thread pyproject.toml
@evan-greenbrg

evan-greenbrg commented Mar 10, 2026

Copy link
Copy Markdown
Collaborator

Confirmation was a bit strange. Using the same AV5 run call. I didn't see an appreciable difference for the presolve, but did see a consistent speedup with the main solve.

With -
dev -> numba

Presolve -
3.9 spectra/s/core -> 3.7685 spectra/s/core

Main solve -
0.4461 spectra/s/core -> 1.7633 spectra/s/core
Speedup: 2.9x

@pgbrodrick

pgbrodrick commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator Author

@evan-greenbrg , which version of python are you on? I'm using 3.12. I ran a repeat test on the above, and reproduced the timings. I also saw that when running 5 different instances, the timings were repeatable within 0.02 spectra/s/core between runs (main solve) and 0.05 spectra/s/core between runs in the presolve. So version differences (including package pinning, python, etc.) is the best I can think of. For what it's worth, I'm also on numba 0.64.0.

@evan-greenbrg

evan-greenbrg commented Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@pgbrodrick I tested on:

Python: 3.10.16
Numba: 0.64.0

I'll try with a python 3.11 or 3.12 install and re-run. Given that you are successfully reproducing and I'm not having any problems or slowdown on this version I'd be in favor of moving forward with it.

Keeping the baseline mlg populated in the class also means we can fall back if need be.

@pgbrodrick

pgbrodrick commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator Author

I could definitely see this being a python 3.11+ difference. I do think I'd be keen to merge this in if others agree.

@evan-greenbrg

evan-greenbrg commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

I just finished running again with Python 3.12. I can reproduce your speed up:

Presolve: 4.4151 spectra/s/core
Main solve: 0.7562 spectra/s/core (reflecting the poor logging, but consistent with the values reported above)

@evan-greenbrg evan-greenbrg merged commit a0d5599 into isofit:dev Mar 12, 2026
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@unbohn unbohn unbohn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pgbrodrick @evan-greenbrg @unbohn