Looks good after first read

@wpoely86: added some extra stuff w.r.t. checking for existing (potentially) hidden modules, and fixed your remarks

I'd like to be able to merge this ASAP, since this is (kind of) blocking other stuff I have queued up.

Can you review this PR again?

Second read, still looks great 👍

Jenkins is happy with this too, so merging in, thanks for the review @wpoely86!

@ocaisa has a good idea to leverage this support for hidden modules, to make modules for stuff that users typically don't care about (i.e., wouldn't load directly themselves) hidden. Examples include libreadline, ncurses, X11 libraries, etc... With this in place, it should suffice to simply move dependencies like these to hidden_dependencies instead of dependencies in the resp. easyconfigs...

I totally vote for this! Using hidden_dependencies instead of dependencies would allow to only present users modules they care about and vastly reduce clutter in "module avail" output.

That, in conjunction with HMNS, would be a big win for end-users.

I'm having second thoughts on installing stuff like ncurses and X11 libs via hidden_dependencies, since that kind of enforces the use of hidden modules on EasyBuild users.

Instead, we should probably add an EasyBuild configuration option that allows to list the software packages that should be installed as hidden modules, somewhat similar to --filter-deps supported now:

% eb --help | grep filter-deps -A 4
    --filter-deps=FILTER-DEPS
                        Comma separated list of dependencies that you DON'T
                        want to install with EasyBuild, because equivalent OS
                        packages are installed. (e.g. --filter-
                        deps=zlib,ncurses) (type comma-separated list)

That gives more freedom imho, avoids updating all existing easyconfig files and reserves hidden_dependencies for the special cases, e.g. inserting GCC as a hidden dependency for icc/ifort.

Thoughts?

Can we not use hierarchy for this? Now, you got MPI where everything ends up in. Maybe split this up in supporting libraries and end user applications? Both will be visible with avail and spider but you can make a clear distinct between in the output of module av so the end users can easily see which programs are installed.

@boegel: I was not sure what you meant, but I think I get it now.

I was also thinking about a configuration flag that would treat hidden_dependencies as regular dependencies, if set. That would still require to update the existing easyconfig files, though.

Overall, I'm ok with your approach, even if it means I'll have to manually specify extra arguments when I install packages, different for each one of them, if I want to use hidden dependencies.

@wpoely86: sure, with a custom module naming scheme you can do that already

The point is that EasyBuild shouldn't force you one way or another, every site will have the policies they want to stick to, EB should allow to implement all of them.

@boegel Although having nice and easy to use defaults is a big plus. Having to write a custom module naming scheme to start with is a big showstopper, at least for me.

@kcgthb: so, it seems we need both a --no-hidden-modules and --hidden-dependencies=X,Y,Z as configuration options?

@kcgthb: the naming scheme remark was w.r.t. installing apps and deps in separate trees, see @wpoely86's suggestion; hidden modules is completely orthogonal to that

@boegel: well, probably. --no-hidden-modules would only applies to easyconfig files that contains hidden-dependencies, and --hidden-dependencies=X,Y,Z only to those which don't.

So yes, that's the most flexible approach. Maybe the most confusing too. :)

@boegel Installing apps and deps in separate trees is also a nice approach. But this would have to be available as a configuration switch (as a provided module naming scheme) to be really usable. I don't think it's reasonable to expect first-time EB users to write their own module custom naming scheme right away to benefit from this apps/deps separation.

@kcgthb: I'd say --hidden-dependencies would complement all easyconfigs, whether they use hidden_dependencies or not

@kcgthb: I don't think providing a configuration switch to differentiate between apps and deps would work. When is something a dep and not an app? That seems a site-specific aspect to me...

Ok for --hidden-dependencies=X,Y,Z, then. That's probably the best approach.
What about a catch-all value, such as --hidden-dependencies=ALL_DEPS or something like that, which would treat all of the easyconfig dependencies as hidden ones?

Aldo you can do a split between end users apps and supporting libraries in the namescheme technically, I don't think it's possible in practice. You have no way of knowing what a easyconfig is? Something in the catagory chem can be a supporting library or a end user program. You would need to add new catagories for it to work.

@kcgthb: w.r.t. the ALL_DEPS: say Python is used as a dependency for something else... You install Python first, and then install something else that uses Python as a dep later. Is Python hidden or not? Or both? I don't see how that would work, a catch-all doesn't make much sense to me.

I agree with @boegel, hidden deps should be used sparely. If you want to present nicer and cleaner to the end user, hierarchical modules are the way to go.

@wpoely86: well, I think actually hiding stuff that users would never load directly, like ncurses, via a hidden module makes sense; I think mixing both a HMNS and hidden modules makes a lot of sense, both are orthogonal to each other imho

What if an end user wants his home made program to compile with ncurses? I wouldn't hide it, just make it less visible 😉

Yeah, well, the optimal solution probably doesn't exist... So, each site should be able to configure EB to make it behave as they want it to, as easily as possible (like @kcgthb already mentioned, not having to dive deep into the framework and write custom Python code, if possible at all).

So that means a naming scheme based on the catagories? Then it should be possible to alter the output of module av to display something better then other.

@wpoely86: picking categories that appeal to everyone is equally impossible... That's just moving the problem imho.

@boegel. Regarding ALL_DEPS and your Python example: when EB checks for existing modules at the beginning of its installation process, couldn't it detect that a hidden module already exists and display a warning about it? Or similarly, when a package is built with --hidden-dependencies=Python and a visible Python module already exists, it could just ignore the --hidden-dependencies directive (I feel like visible modules should take precedence over hidden ones).

@kcgthb: I certainly can do that, both cases. I guess the question there is: should EB complain (and fail) when it detects an 'inconsistency' like that, or implicitly pick up what is there (e.g. pick up GCC/.4.8.2 when GCC/4.8.2 is specified, and vice-versa)? I'd say that's (yet another) configuration switch, and the default should probably be complaining and failing (to avoid surprises). Starting strict and allowing to loosen it up is better than the other way around imho.

I think that hiding is a bad idea in general. First hidden modules are only hidden from avail and spider and not from module list. We all have novice users but not all users are novice. When you hide things, you hide them from users who want to know its there. You are going to get questions about say ncurses/.4.2.1 which wasn't on the avail list but is listed with the modules your users have.

The only time I think that hiding is a good idea is when you have to install something that staff and maybe friendly users to test before making available for general use.

@rtmclay: don't you agree that hiding a GCC module that purposely does not extend $MODULEPATH included in a module hierarchy makes sense? See easybuilders/easybuild-easyconfigs#1014 (comment). Users shouldn't be loading this one directly, since it won't yield the result they expect (i.e., other modules turning up)...