Also cc @rgrinberg for the Dune part.

Naive question again, sorry: why are we using threads/META and not threads/META.threads, and same for compiler-libs?

Naive question again, sorry: why are we using threads/META and not threads/META.threads, and same for compiler-libs?

Yes, sorry I should have been more explicit. This is a Findlib convention: the package name for a META file is taken to be the extension (if it exists) or, if there is no extension, it is the name of the parent directory.

This is a Findlib convention: the package name for a META file is taken to be the extension (if it exists) or, if there is no extension, it is the name of the parent directory.

not quite true. There are only two possible location and name for META files currently supported by ocamlfind:

• ${SITELIB}/META.<your-lib>
• ${SITELIB}/<your-lib>/META

any other combinations are ignored. The value of ${SITELIB} is given during the configuration of ocamlfind but it is typically:

• ${prefix}/lib in opam
• ${prefix}/lib/ocaml in system installations of ocamlfind (e.g. debian, …)

This is a Findlib convention: the package name for a META file is taken to be the extension (if it exists) or, if there is no extension, it is the name of the parent directory.

not quite true. There are only two possible location and name for META files currently supported by ocamlfind:

• ${SITELIB}/META.<your-lib>
• ${SITELIB}/<your-lib>/META

There is also OCAMLPATH which is interpreted by both ocamlfind and Dune.

I’m not sure why we need to add +/stdlib to searchpath. Why not simply do exactly as ocamlfind already does and add the ${SITELIB} distinction directly in the compiler?

Are you suggesting to install the compiler libraries directly inside ${SITELIB}? I guess this would work but it would mean changing the installation layout (which is another Big change). We still need a story for non-ocamlfind users, so I think that the current approach will need to be kept, if only for them.

Are you suggesting to install the compiler libraries directly inside ${SITELIB}? I guess this would work but it would mean changing the installation layout (which is another Big change). We still need a story for non-ocamlfind users, so I think that the current approach will need to be kept, if only for them.

mh no no, this wouldn’t require any change in the installation layout. The only thing that we would need to be installed compared to trunk would be simply:

• ${SITELIB}/bytes/META
• ${SITELIB}/compiler-libs/META
• ${SITELIB}/dynlink/META
• ${SITELIB}/ocamldoc/META
• ${SITELIB}/stdlib/META
• ${SITELIB}/str/META
• ${SITELIB}/threads/META
• ${SITELIB}/unix/META
• ${SITELIB}/findlib.conf

exactly as ocamlfind currently does. The content of the META files already retirect to the correct directories (see what is installed by ocamlfind). This does not change anything for non-ocamlfind users.

This does not change anything for non-ocamlfind users.

OK, I understand your suggestion now. We still need to do something for non-ocamlfind users, right? Installing in the compiler directory seems to be the most natural alternative in this case...

We still need to do something for non-ocamlfind users, right? Installing in the compiler directory seems to be the most natural alternative in this case...

I don’t understand. In my scenario nothing else changes so why do we need to do something else?

I don’t understand. In my scenario nothing else changes so why do we need to do something else?

Because META files are consumed by Dune (even if you don't use ocamlfind). As I see it, your scenario only addresses the needs of users using ocamlfind, but for other users we still need to install these META files somewhere so that Dune can find them, right?

Because META files are consumed by Dune (even if you don't use ocamlfind). As I see it, your scenario only addresses the needs of users using ocamlfind, but for other users we still need to install these META files somewhere so that Dune can find them, right?

mmh, sorry I don’t understand either. dune handles findlib.conf and META files correctly afaik.
If you are implying that it doesn’t then it looks like a bug in dune.

In any case, dune already has builtin dummy definitions for the compiler packages (which are taken over the META files since ocaml/dune#4946) so it doesn’t matter for dune anyway.

Again, with my proposal, this would be exactly equivalent to what ocamlfind already does. Dune and all the other ecosystem parts work pretty well as-is.

In any case, dune already has builtin dummy definitions for the compiler packages (which are taken over the META files since ocaml/dune#4946) so it doesn’t matter for dune anyway.

Well, one of the objectives of the present PR is for Dune not to have to keep its own set of built-in META files which causes issues now and then. It doesn't feel right to make something that will only benefit ocamlfind users...

Well, one of the objectives of the present PR is for Dune not to have to keep its own set of built-in META files which causes issues now and then. It doesn't feel right to make something that will only benefit ocamlfind users...

again, AFAIK dune already handles META files correctly, and even if it is missing something this would only be required to be fixed whenever you wish to remove those dummy builtins, which already requires patching. So this doesn’t change anything.

I’m not seeing why this would only benefit ocamlfind users. This benefits everyone using the tried and tested (19 years) solution.

I’m not seeing why this would only benefit ocamlfind users. This benefits everyone using the tried and tested (19 years) solution.

I feel we are talking past each other. I agree with your suggestion to install META files directly into ${SITELIB} for ocamlfind/opam users. The question I am concerned with is where to install the META files for those users who do not use ocamlfind nor opam. Where should that be, according to your proposal?

Where should that be, according to your proposal?

Same as for "ocamlfind/opam users": ${SITELIB} (i.e. wherever it was configured during the OCaml installation)

${SITELIB} would be accessible in a new site-lib: (or whatever name the dev team prefers) field shown by ocamlc -config and by default, when undefined, ${SITELIB} as defined by ocamlfind currently is located in: ${prefix}/lib/ocaml/site-lib, this can be changed of course.

Overall, it simply moves things currently done in ocamlfind to the compiler. For example:

• in opam it would simply be passed as argument to the configure script of the compiler:
  • ./configure --prefix "%{prefix}%" --site-lib-dir "%{lib}%"
• in debian/archlinux/..., same idea:
  • ./configure --prefix /usr --site-lib-dir /usr/lib/ocaml
• in freebsd/..., same idea again:
  • ./configure --prefix /usr/local --site-lib-dir /usr/local/lib/ocaml/site-lib

Thinking about it more, I don’t think we even need the findlib.conf file, this would still be installed by ocamlfind separately. I’ll edit my previous comment mentioning it.

Same as for "ocamlfind/opam users": ${SITELIB} (i.e. wherever it was configured during the OCaml installation)

Sorry, I still don't understand what is the use of doing this for people that do not use ocamlfind/opam. Dune would still have to be patched to look inside this new ${SITELIB} directory to find the compiler-installed packages.

Overall, it simply moves things currently done in ocamlfind to the compiler. For example:

As mentioned, something along these lines in the case of an ocamlfind/opam setup seems fine, but I don't see the point of doing it in a non-ocamlfind context: it just creates yet another directory hierarchy with mostly empty directories. Adding + to Dune's OCAMLPATH just seems simpler.

For the benefit of readers struggling to follow the back-and-forth (like myself :)), I try to summarize the state of the discussion so far. The question is: where should the META files for the compiler libraries be installed?

@kit-ty-kate proposes the addition of a new configure-time flag --site-lib-dir=$SITEDIR. When this flag is used, make install installs META files in $SITEDIR following following standard ocamlfind/opam conventions, ie:

• $SITEDIR/stdlib/META
• $SITEDIR/unix/META
• $SITEDIR/dynlink/META
• etc

This flag would be used by opam when installing compiler packages and on the various Linux distribution packages which use ocamlfind. Its main benefit is that none of the existing tools interacting with META files needs to be modified (ocamlfind, dune, opam). This proposal seems consensual.

There is also the question of what to do when --site-lib-dir is not used. This is the case when ocamlfind/opam is not used. META files are still useful because they are consumed by dune. Where should they be installed? @kit-ty-kate proposes to proceed as before with $SITEDIR set by default to $PREFIX/lib/ocaml/site-lib. Concretely this means that the META files would be installed as follows:

• $PREFIX/lib/ocaml/site-lib/stdlib/META
• $PREFIX/lib/ocaml/site-lib/unix/META
• $PREFIX/lib/ocaml/site-lib/dynlink/META
• etc

There is another possibility which is to install the META files next to the library files themselves:

• $PREFIX/lib/ocaml/META.stdlib
• $PREFIX/lib/ocaml/META.unix
• $PREFIX/lib/ocaml/META.dynlink
• etc

In both cases, Dune will need to be patched to look inside $PREFIX/lib/ocaml or $PREFIX/lib/ocaml/site-lib for the newly installed META files. Personally I find the introduction of the directory hierarchy $PREFIX/lib/ocaml/site-lib just to maintain an uniform installation scheme between ocamlfind and non-ocamlfind cases a bit jarring, which is why I am slighly in favor of installing (in this non-ocamlfind context) META files next to the libraries themselves, and not in a new site-lib subdirectory, but more discussion is necessary to make a decision on this point.

There is also the question of what to do when --site-lib-dir is not used.

Why isn't --site-lib-dir simply required with OCaml 5 onwards (with a sensible default)? Since it's a new version of OCaml, OS package managers can specify their OS-appropriate version, so we can assume it's always available. Is the problem that dune doesn't have a default site-lib-dir to look in at the moment (or needs to be pointed to one by findlib.conf)?

Why isn't --site-lib-dir simply required with OCaml 5 onwards (with a sensible default)?

Sounds fine, but the question is what that default should be. In the case of a non ocamlfind/opam-installation one natural default is $(ocamlc -where). If we do that, there is the issue of which directory layout to use: ie whether to install $(ocamlc -where)/stdlib/META or $(ocamlc -where)/META.stdlib (I tend to prefer the latter).

Is the problem that dune doesn't have a default site-lib-dir to look in at the moment (or needs to be pointed to one by findlib.conf)?

This is another aspect of the issue. Dune does not have a notion of "site lib" and does not read findlib.conf, but will honour the OCAMLPATH variable (in general Dune has been moving towards depending less and less on ocamlfind). So it will have to be adapted in order to look for META files in whatever default we choose for --site-lib-dir in the case where ocamlfind/opam are not being used.

I'm sorry but I have difficulty in following the discussion here. I don't understand:

1. Why should the ocaml package be concerned about site-lib, findlib.conf or OCAMLPATH ?
2. Why empty directories should be created just to put META files in them.

I think the META files should simply be installed in the directory where the corresponding library archives are found. Which means in ocamlc -where, ocamlc -where/compiler-libs and ocamlc -where/threads.

Given how packages get their name in ocamlfind, they will also need to be of the from META.$(libname) so that the existing names (dynlink, str, unix, threads, stdlib) are preserved.

I'm not sure I see the issues arising from such a simple implementation.

All of this of course unless upstream eventually wants to bite the bullet to properly isolate its libraries in directories and possibly namespace them under the ocaml name as was suggested in the library linking proposal :-) (a transparent compatibility story could easily be added by having META files for the old names and requireing the new names).

I'm sorry but I have difficulty in following the discussion here. I don't understand:

1. Why should the ocaml package be concerned about site-lib, findlib.conf or OCAMLPATH ?

If I understood correctly, @kit-ty-kate's point was that if we just install them in $(ocamlc -where) (my initial proposal), then ocamlfind (and perhaps opam?) will need to be adapted to find these META files because it does not currently look inside $(ocamlc -where) for META files. So it may be better from a compatibility perspective to install them inside an existing "site lib" in this case.

This sounds somewhat reasonable to me, but of course for cases when there is no pre-existing "site lib" I suggest to do as you suggest, namely to install them in $(ocamlc -where), next to the libraries themselves (most of the above back-and-forth appears to be a disagreement around this point...).

2. Why empty directories should be created just to put META files in them.

Agreed, I don't think we should do this.

I'm not sure I see the issues arising from such a simple implementation.

This is what @nojb is proposing in this PR I believe. However the issue I was pointing out is that ocamlfind does not work this way and would have to be changed (same for dune and virtually any other build systems relying on META files).

My proposal is to instead import what ocamlfind is currently doing. This way there are no unforceen breakage and annoying patches to make (only dune has to be patched, but will happen regardless of the solution chosen, so it doesn’t matter)

However the issue I was pointing out is that ocamlfind does not work this way and would have to be changed (same for dune and virtually any other build systems relying on META files).

Sorry to add more to the confusion. Indeed I was mislead by the documentation ocamlfind I linked to:

The outermost variable assignments and additions belong to the main package. The name of the main package is not defined within META; it is either the name of the directory containing META or the suffix of the META file (if the name of the META file is formed like META.name).

Which unfortunately (I tested) doesn't seem to be true.

I'm sure this is not going to be popular, I'll just mention it. Why not take the opportunity to introduce the ocaml.* prefix ? That way you can just install a single meta file in $(ocamlc -where)/META. We can then add warnings to existing META files on the old names to let people migrate to the new ones and the day (one can dream) we get some form of namespacing/library thing, that bit will already be in a better shape (and no system should require changes).

I'm sure this is not going to be popular, I'll just mention it. Why not take the opportunity to introduce the ocaml.* prefix ? That way you can just install a single meta file in $(ocamlc -where)/META. We can then add warnings to existing META files on the old names to let people migrate to the new ones and the day (one can dream) we get some form of namespacing/library thing, that bit will already be in a better shape (and no system should require changes).

In theory i like the idea, however in practice sadly the same problem arises because ${SITELIB} isn’t linked to $(ocamlc -where) in any way currently and things need to be changed (in slightly different ways than @nojb’s proposal because e.g. you need to support ${SITELIB}/META when ${SITELIB} = $(ocamlc -where))

you need to support ${SITELIB}/META when ${SITELIB} = $(ocamlc -where))

Right but IIUC this is a matter for system packagers to do a mv META META.ocaml no ?

It feels as if the discussion is diverging. So far I see the following three options:

1. Install META files in $(ocamlc -where) next to the library archives (@dbuenzli's suggestion and the current implementation of the PR). ocamlfind needs to be adapted to look in there for META files.
2. Add a configure-time flag --with-site-lib=$SITELIB. When used, install META files in $SITELIB according to existing ocamlfind conventions (@kit-ty-kate's suggestion). If the flag is not specified, install them in $(ocamlc -where) as in 1).
3. Not do anything for the time being and revisit the issue in the future.

Opinions? Personally I am starting to feel that perhaps 3) is the best right now :)

It feels as if the discussion is diverging.

I thought it was actually converging :–)

Opinions?

Well if you are going to do something, do something that works according the way things works for the existing tools.

I mean 1) only makes sense if you consider's ocamlfind's current behaviour a bug (reading the docs, you could actually) and the default in 2) doesn't really make sense in practice.

I initially was not so hot on introducing the ocaml.* prefix but I'm getting more and more convinced we should. Since finally ocaml's library install structure starts to act like (normalized I would say) ocamlfind packages do. I have the impression that not doing so here simply adds more churn in the future.

The more I think about this PR, the more I would like it to bite that bullet, so that ocaml makes the first steps towards following a regular opam/ocamlfind package structure. This would at least establish the library names in a forward looking way.

I have written a fair amount of eco-system tool and build systems and I think there's not a single instance in which I didn't have to make special cases and introduce hacks for ocaml's package and distributed libraries. Don't worry, you will remain our beloved cathedral even if you conform a bit more.

Also it actually makes your task and maintenance simpler: a single META file to maintain that describes all the libraries you distribute.

I have rebased the PR;and updated the META file to use the new installation directory for libraries.

The installation path for META files is now configured by a new SITELIBDIR configuration variable which was added by @shindere . The default value is for now $libdir, since opam was already using a custom setting for ocamlfind's sitelib.

Patching dune(ocaml/dune#5824) and ocamlfind(ocaml/ocamlfind#45) to use those new META files seems quite straightforward.
And after a short test, at least opam switch seems to work fine with this change (after a small tweak ocaml/opam-repository#ocaml/opam-repository#21468 to the compiler opam files).

There's a couple of details I'm still checking, but the only major suggestion I have is for findlib itself - it will be easy to have findlib's configure script require that it's given the same sitelib as the compiler was for OCaml 5 (i.e. findlib's configure script will fail if it detects OCaml and doesn't detect, say, the stdlib META file)

Following both a closer look at this, and an offline discussion with @Octachron and @shindere, we propose reverting to something much closer to @nojb's original suggestion. Concretely, this means reverting the addition of SITELIBDIR and always installing the META files to subdirectories of OCaml's lib directory (+stdlib/META, +unix/META, etc.). The change becomes the single commit dra27@744da2c and requires a single small patch to Dune (see dra27/dune@fec8a08), and a relatively small patch to ocamlfind (previewable in dra27/ocamlfind#1).

The rationale for this is catering for opam's ocaml-system and also opam-less use of Dune, which both require a solution that works with an arbitrary pre-existing installation of OCaml. ocaml-system allows an OS-installed (or otherwise non-opam-installed) OCaml to be used with an opam-installed ocamlfind. This works in 4.x because only ocamlfind installs META files to a single site-lib directory, and an opam-controlled ocamlfind will install META files in an opam-controlled site-lib directory. Similarly, dune can be used without opam or ocamlfind with OCaml 4.x because of its "internal" META files. With a configurable SITELIBDIR, for an arbitrary installation of the compiler, we may have the right META available somewhere, but we are left without reliable means to locate them, and more importantly we cannot move them nor assume that we will be able to install other META files alongside them.

In other words, supporting both of these existing use-cases (which are, incidentally, really used "in the wild") leads us to an ownership problem with the META files. In both instances, we need either to able to guarantee where the META files are for a 5.x compiler, or we have to be able to probe where they are (by reading +Makefile.config or by being able to parse the location from ocamlc -config). We also need to assume that the compiler META files might be located in a non-writable directory. Given that both Dune and ocamlfind already need updating for this change, it is simpler instead that the compiler completely owns its META files and requires them to exist under the Standard Library directory. That of course doesn't preclude OS packagers and/or opam choosing to copy them elsewhere, but, luckily, the co-evolution of ocamlfind, Dune and OCaml leaves us with a mechanism to keep this simple, too. The two linked patches for Dune and ocamlfind utilise the pre-existing mechanism for the default value for OCAMLPATH which means that in all use-cases, the compiler's META files are searched and then the normal site-lib directory is searched. Users will notice no change, as it's only those two tools which need configuring.

@nojb are you fine with the current state? Trying to untangle the web of contributions, @dra27 and me thought that it might be best if you approved the final state and while we consider that @dra27 (and me?) reviewed the opam and ocamlfind part of the PR.

@nojb are you fine with the current state? Trying to untangle the web of contributions, @dra27 and me thought that it might be best if you approved the final state and while we consider that @dra27 (and me?) reviewed the opam and ocamlfind part of the PR.

Yes, I am fine with the current state, thanks for all the work!

Cherry-pick to 5.0 970af7d

There are lone + in the directory variables. What does that mean? Now that I actually look for documentation about it, I notice that neither a plain ^ nor a plain + is documented. I looked in the ocamlfind and ocaml-library-standard repos.

(the plain + breaks my existing META parser, which expects something after the plus sign)

In other words: what will break if the + is not there? The SUSE ocaml package contains /usr/lib64/ocaml/threads/META:directory = "^" since Nov 2019.

Furthermore, is the +unix actually needed in /usr/lib64/ocaml/unix/META:3:directory = "+unix"?
It is my understanding that somedir/META implies a +somedir already.

What does that mean?

+ and ^ are synonyms and they stand for the OCaml standard library directory.

what will break if the + is not there?

ocamlfind will look for archives in the package directory (the directory containing META), and will fail if the archive is not found there (this is the case for stdlib).

Furthermore, is the +unix actually needed in /usr/lib64/ocaml/unix/META:3:directory = "+unix"?

I believe it is not actually neeeded. Indeed, the directory variable is only required for stdlib as the archive is not in the same directory as the META file. These variables are probably remnants from when these META files lived outside the compiler; they could be omitted now (except in the case of stdlib).

Yes, I'll add the removal of directory from most files to the set of updates.

Alas, the internet doesn't quite have the archives to be able to see the precise history, but ^ and + have meant the same thing in ocamlfind for at least 19 years! As far as I know, the ^ has always been findlib-specific - I've preferred its being excised, FWIW (the meaning of + in -I is documented in OCaml's manual)