This PR is the first and main PR of a brave (in the Apple sense) series of changes which seek to separate the components in OCaml's $LIBDIR. This PR specifically moves the Dynlink, Str and Unix libraries into sub-directories of $LIBDIR. The idea of the change was provisionally agreed at the last developers' meeting.

The motivation for this change is hygiene in build systems: at present, it is very difficult to impossible for Dune (for example) to ensure that a library specifies its dependency on the Unix library.

The change itself is quite simple, and in the first commit. A small tweak to two Makefiles ensures the libraries are installed to their new locations. Any previous artefacts are removed (cf. #1724, #1728 and #10301, noting that the removal of artefacts is "precise" w.r.t. basenames).

That change on its own is very breaking, of course.

The rest of the commits relate to backwards compatibility. The compilers, toplevel and debugger will all automatically add the required include path to the load path, but display an alert for each directory as this happens. For example:

$ ocaml unix.cma
File "_none_", line 1:
Alert ocaml_auto_include: OCaml's lib directory layout changed in 5.0. The unix
subdirectory has been automatically added to the search path, but you should add
-I +unix to the command-line to silence this alert.
OCaml version 5.0.0+dev0-2021-11-05
Enter #help;; for help.

or similarly with:

$ ocaml
OCaml version 5.0.0+dev0-2021-11-05
Enter #help;; for help.

# type t = Unix.file_descr;;
Alert ocaml_auto_include: OCaml's lib directory layout changed in 5.0. The unix
subdirectory has been automatically added to the search path, but you should add
-I +unix to the command-line to silence this alert.
type t = Unix.file_descr

In implementing this alert, I've attempted to avoid penalising code which has specified the appropriate include directories, especially as we know that the load path is a relatively "hot" part of the build for some users. The mechanism I've gone for is to add an extra attempt around Load_path.find and Load_path.find_uncap for the case when there is an actual miss on the load path. In almost all cases, a miss on the load path is fatal. The exceptions are #remove_directory in the toplevel and gaining inlining information from .cmx files when linking in ocamlopt, so at less critical stages. The toplevel, compilers and debugger add "+dynlink", "+str" and "+unix" to a "secondary" load path which is lazily scanned on the first miss on the normal load path. If the file being searched (say unix.cma and unix.cmi) is found, then that directory's entries are promoted to the main load path, and the fact that this took place is recorded. From the user's perspective in, say, the toplevel what happens at #load "unix.cma" is as if the user issued #directory "+unix" and then tried the #load again.

Load_path is very low-down the dependency graph of compiler-libs, so rather than displaying alerts itself, it simply maintains a list of directories which have been automatically added. This list is queried by functions added with the other alert functions in, ahem, Location.

The initialisation code is a duplicated between the debugger, driver and toplevel. It's not huge, but I'm not sure exactly where to put a function to do that - how averse are we to adding modules to ocamlcommon?

Another option might be to install the files in both places, but this isn't a silver bullet either - including the new directory can cause warnings in some tools (ocamlfind doesn't like finding .cmi files in multiple directories, for one) and it seems to be harder to record the need to display an alert without altering main load path data structure, which is a much more intimidating prospect than altering an error path.

Very similar work was previously attempted and abandoned in #1569. This work differs both in implementation and timing in a few ways which hopefully means we can reach a consensus on merging it this time. The key differences:

• OCaml 5.0 already breaks quite a few older packages, which means that this change is not as sensitive as it was in 2018.
• We have superior testing systems in place to assess the impact of this change. For OCaml 5, accepting that both ocamlfind and dune require patching, only one other package is broken in the whole of opam-repository by this PR, and that's an old release of that particular package, the current release using Dune and being unaffected.
• No changes are made to compiler-libs.
• Rather than implementing new compiler flags, this PR displays an alert

I have tested this change quite extensively on 4.14 - in many cases, the packages affected are already lying to their build systems. Dune and ocamlbuild users, for example, should not use Unix without having unix in their libraries stanza or use_unix in their _tags file. Even packages building directly with ocamlfind should include -package unix in the command-line. These packages can be fixed regardless of whether this PR is merged. Given a flurry of PRs now with this change (these packages can merge these updates regardless of whether this PR is accepted), I imagine that by OCaml 5.2, the number of packages still displaying this alert will be close to zero and the entire mechanism can then be reverted - at which point this change lives entirely in the build system.

There are other packages which contain commands of the form ocaml unix.cma which want to become ocaml -I +unix unix.cma - PRs are opened for these as well.

There are three possible pieces of future work which I haven't done yet, mainly because they are more philosophical (separating the otherlibs out mitigates actual issues):

1. C stubs for otherlibs are still installed to $LIBDIR (this is already the case for systhreads, so this is partially done to stay consistent). This could be trivially fixed for all these, but it could be viewed as separate. Note that at present we're saying that -L /usr/lib/ocaml is all you need to give to link in C. I can't remember the reference, but there's at least one application in the wild which links manually with our C stub libraries, because I remember dealing with a bug report relating to it...
2. std_exit.cmo and std_exit.cmx remain, despite the fact these are really part of the compiler, and not the Standard Library. The compilers impose a requirement that a standard library replacement implements a function called do_at_exit, but the actual implementation of std_exit is part of the compiler, not the stdlib. Indeed, in many ways there's no need for a module called Std_exit at all, we could generate a temporary file.
3. The runtime and stdlib still share the same directory. This is also a more principled matter, than actually causing issue: there's an open question for whether the runtime should go in +runtime or the stdlib in +stdlib. FWIW, I would advocate the latter - lib/ocaml now reflects the files of the OCaml runtime and compiler with the sub-directories all being libraries. But that can readily be for another day and another release...