@becker33 can you comment on this?

The failing test says:

Test that the '-r' option to the load command prepends dependency prefix

but -r is not used in the test, -r has no effect in the command (see #25731), and the test can't hold with just --only=package or --only=dependencies options, you can't specify --only=package,dependencies on the command line, since that's an invalid option according to argparse.

For more context see #25723 (comment)

@haampie Results (minutes:seconds) for four trials of the spack load -r huge_deps_package on one of our nodes are below. As it stands, it still looks like the hacky fast-path is worth vendoring for us if we can't find a way to upstream it. Happy to try other things though, but environments are probably not going to fly in the short-term at least (changeover burden, etc.).

This is confusing... it's strictly slower? It's also strictly doing less work than develop...

For me after spack install py-flake8:

In [1]: from spack.main import SpackCommand
In [2]: load = SpackCommand('load')
In [3]: %timeit load('--sh', 'py-flake8')

Develop: 695 ms ± 4.19 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#25732 + #25755: 97.4 ms ± 1.39 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

That's 86% faster...

Curiously though I don't see much speedup from the command line:

$ time spack load --sh py-flake8 > /dev/null
real	0m1.427s

$ time spack load --sh py-flake8 > /dev/null
real	0m1.333s

Looks on par with develop for the spack load -r to me. The first trial is sometimes slow for whatever reason (regenerating cache when first connected to the chained instance when I set that up or something?).

Anyway, it was pretty clear for me that I wasn't getting much benefit for the recursive load of ~100 specs with the current situation at least.

And yeah, we're using command line (tcsh)--so if that command gets called say 100 times, it is roughly equivalent right--you're going to end up at over a minute typically.

Right... maybe the packages you're loading specify run type deps but really are only link type deps? (i.e. the dependent does not need <transient dependency prefix>/bin in its path?)

At any rate, I'm confused why load('--sh', 'py-flake8') is fast but spack load --sh py-flake8 almost equally slow. Maybe some caches get destroyed indeed...

A focus on linking/"including" sounds like a reasonable description of what we have yeah--after all, at least on our end, we really just want CMake to be able to find what spack built and link things in/compile header-only libs, etc., rather than directly executing those dependencies.

Edit: although we do need some binary paths accessible I think--like spack-built Python for example.

Okay, so what I've found is: there's 0.6s (!) difference between querying spack load py-flake8 vs spack load /iv36zrg, and further setup_[dependent]_run_environment of python/package.py is rather slow, also we do seemingly redundant calls to set_module_variables_for_package which imho only makes sense for build environments. All in all that gets me to spack load /iv36zrg in 0.5s, which is still slow, but also includes python startup time, which is about 70% of that according to the profiler.

So, disambiguate spec is the bulk of the time.