Partly solves MPR7362.

I think (but I have not discussed this recently) that the reluctance to expose interface and implementations of the compiler-libs content comes from the fact that these modules are not documented and that no stability guarantee is provided for their users. The OCaml compiler maintainers often have a culture where the good, stable, exposed interface should be documented, and the private, non-stable, dangerous interface is un-documented -- and whether or not there is documentation can be understood as a message to indicate the intent of encouraging use and trying to provide stability.

I would personally agree that even the interfaces that are not meant for wide use and that are unstable could and should be documented. However, there is also a real concern with lack of clear information of what should be considered external/stable and internal/unstable. Finding ways to address this concern could be a useful way to push the proposed changes further.

For example, maybe it would be a good first step to add to the .mli of modules in compiler-libs which are undocumented and unstable an explicit comment reminding their readers that those files are internal to the compiler and that no stability guarantee is offered for them.

@gasche I have no opinion whether the compiler-libs interfaces should be installed or not. But I'd really like that mli and cmti files are at least consistently installed or not installed for all the libs of the distribution, so that odig can provide a consistent view of the libs API to the users whether it is using ocamldoc or odoc.

I understand your request from the point of view of the tool, I was pointing out that there were clear social reasons why the stdlib files and the compiler-libs files are handled differently, this is not merely a matter of an unexplained and unjustified inconsistency.

I believe that one needs the mli and the ml files of compiler-libs if one really wants to program against this library. Currently users need to download the OCaml sources separately because neither opam nor debian install the those files.

The OCaml reference manual clearly says "the exported front-end interface [...] does not provide any backwards compatibility guarantees." From my point of view, this warning is completely sufficient.

For moving forward, it would be nice to find somebody who is able and willing to decide about what needs to be done to merge this pull request. @gasche could you do this?

I can put a comment

(** This interface follows the evolution of the OCaml language and implementation and is likely to change in the future. There is no guarantee about compatibility with future releases. *)

into those files referenced from the OCaml manual, and a comment

(** This is an OCaml internal interface that is likely to change in the future. Usage is discouraged. *)

into all other files. But before doing so I would like to reach an agreement about these comments and the the other conditions for this pull request.

I believe that one needs the mli and the ml files of compiler-libs if one really wants to program against this library.

I can see why this applies to the .mli files since they (theoretically) contain documentation, but why put the entire source of of the compiler in people's lib directory?

but why put the entire source of of the compiler in people's lib directory?

The ml files are installed for many libraries, see for instance the stdlib or lablgtk2.

The real reason is that the documentation in the mli files is not sufficient. It is, for instance, necessary to call Location.init before Parse.implementation. You can find out about this however only by looking at the sources.

The ml files only add 4.5MB on top of 70MB of the compiled modules and 1.2 MB of the mli files.

@hendriktews I think this is actually intentional.
As said earlier, as long as we do not commit to an official API for the compiler libraries, we cannot explicitly encourage people to use them directly. The cmi's are there so that one can compile compiler extensions against them, but no to allow people to start writing them without understanding how the compiler works (to some extent). So it seems reasonable to require that people first download the source files, or look them up on github.

This said, as you point out, this situation is not ideal when one wants to use tools to browse the sources. I already had this problem a long time ago when I wrote ocamlbrowser, and used it for the compiler itself. We could maybe add an extra target to install those files, but just for developers who need them. For others I'm afraid this would send the wrong message.

For moving forward, it would be nice to find somebody who is able and willing to decide about what needs to be done to merge this pull request. @gasche could you do this?

I would expect either @damiendoligez or @xavierleroy to have strong opinions on this.

It would be nice to have the .ml files for otherlibs installed too, if that could be considered at the same time.

With upcoming OPAM 2.0 (compilers as packages), downloading the sources of the compiler is as simple as:

opam source ocaml.4.03.0

and you will find an archive 4.03.0.tar.gz in the directory.
If the reason for installing the sources is to avoid downloading the sources, it is a bit weak now.

The "downloading the source is hard" argument was always weak, but I find @dbuenzli's remark compelling: whenever we install the .cmi and .cmti we should also install at least the .mli because some tools need it.

I'm much less enthusiastic about installing the .ml files: if you want to look at the sources you should look at them in their natural environment, the whole sourcetree.

I don't think we should expose direct sources (.mli for interfaces) in the installation directory. These files may use syntax extensions, preprocessor directives, etc. that will make them unusable by any tool relying on them. The idea should be for these tools to use the .cmti files, that are supposed to contain both types and comments originating from the source. If these files are not usable by the tools, then we should fix them to provide the information needed by the tools, instead of using a partial solution that is bound to fail at some point.

I admit my use-case is obscure. I have a program which uses compiler-libs to directly interpret the OCaml AST.

feast:ocaml-i john$ ./ocamli -e "1 + 2 * 3" -show-all
    1 + 2 * 3
=>  1 + 6
=>  7

feast:ocaml-i john$ ./ocamli -e "List.fold_left ( + ) 0 [1; 2; 3]" -show
6

Now, this is fine on any computer with OCaml installed, because list.ml is installed, so the definition of fold_left is available. It would be nice to have bigarray.ml, unix.ml etc. also installed, since I want ocamli to run on any computer where OCaml can run. Only genuinely external libraries would need special treatment. I hope one day, of course, to persuade all library authors to install their source...

I don't think we should expose direct sources (.mli for interfaces) in the installation directory.

From an odig perspective, I won't need these once odoc is mature enough, but meanwhile I'd suggest to keep them so that people can have the standard lib in their doc via the ocamldoc generation path.

If I can make MPR7362 more precise from a cmti install perspective here are the points:

1. All compiler-libs cmti files are currently installed, so their interface is actually exposed. You might want to change that.
2. otherlibs systematically (IIRC) lack cmti files. This should really be changed.

My 2 cents: I don't buy the arguments that compiler-libs should somehow remains "hidden". It is widely used and is required for a lot of nice and important tools which could not be maintained outside the core distribution otherwise. The lack of API stability guarantee or nice upfront design should not be an excuse to avoid documenting it (and polish it). (Some parts of compiler-libs are actually not too badly documented, such as Parsetree, Ast_mapper, Ast_iterator.) I've nothing against adding more disclaimers, perhaps even in each .mli, about the lack of stability guarantee (and why not some call for helping with improving the doc?).

As for which files to install: I've no strong opinion on it; personally, I'm perfectly fine browsing .mli files from upstream source packages but for easier accessibility, it is rather clear that the current situation is quite bad on the documentation front: no decent way to get cross-referenced documentation for all installed packages. This is an important topic and I'd be tempted to follow the suggestions of people putting efforts into improving this situation (i.e. @dbuenzli). If the community ever produces a better tool than ocamldoc and settles on it, one might always revert later and stop installing .mli files if .cmti are enough to get good documentation, which is not the case today.

For the record, I fully agree with Alain's position above.

@johnwhitington I don't think that there is any issue with treating otherlibs/ stuff like stdlib/ is handled today, so I would be comfortable merging a pull request that does that. It is for the compiler-libs part that I think we need a stronger consensus first.

My motivation for this pull request is not that downloading the sources is hard. That has never been hard and the upcoming "opam source" does not really make it simpler IMHO.

The motivation is convenience: For many library functions, when I type their name, the declaration and the documentation in the mli file are a key stroke away. If that leaves any question open or if I am just curious, I can look at the sources with another key stroke. I have not yet seen a convincing argument why this should be different for compiler-libs.

compiler-libs is described in the OCaml manual, there are many tools using it already. Many module interfaces are contained in the documentation in a hidden way (eg, Ast_invariants, Clflags, Identifiable). I don't think that installing the mli or ml files would encourage anybody to do anything on top of what they planed already.

Anyway, I believe the discussion is far too long already for such a simple change. As I wrote before, I am happy to add the disclaimers. I am also happy to create a different install target for compiler-libs and/or the mli and/or the ml files.

Have you ever tried ocp-browser:
https://www.typerex.org/ocp-index.html#ocp-browser

If we install the sources for the compiler for convenience, then we should do it for all contributed libraries for exactly the same reason. Every OCaml switch is already big (between 200 MB to 1 GB), which makes having a lot of them become very space consuming: a good OCaml dev should make his contributions work on as many OCaml versions as possible, which means having at least all versions since 3.12.1 installed... What will happen if we double the space requirement by adding sources, just for convenience ?

I think a better approach would be for you to create an OPAM package ocaml-sources, with the same sources as the compiler, but which installation procedure would copy the sources in the ocaml sub-directory. That way, your problem would be solved, without impacting other users.

Now, this is fine on any computer with OCaml installed, because list.ml is installed, so the definition of fold_left is available. It would be nice to have bigarray.ml, unix.ml etc. also installed, since I want ocamli to run on any computer where OCaml can run.

That's a pretty specific use case and I don't think it justifies installing the .ml files of everything for everyone.

@hendriktews here's my recommendation: install otherlibs .cmti files and add an install target for .ml files but don't install them by default.

@damiendoligez OK - and the mli files of compiler-libs go into the default installation target?

@hendriktews

and the mli files of compiler-libs go into the default installation target?

Yes, that makes sense.

I updated this PR, taking the comments into account. By default it installs the missing mli, cmti and cmt files. To get some of these cmti files, I added option -bin-annot for ocamldoc and tools. There is an additional target for installing compiler-libs ml sources.

In detail: I looked a bit more systematically for missing cmti files and saw that apart from otherlibs, also

• ocamldoc/odoc_info.cmt
• ocamldoc/odoc_info.cmti
• profiling.cmt
• profiling.cmti
• topdirs.cmti

are missing. I am quite sure that we want to install odoc_info.cmti but for the others I don't really know. If these files were deliberately omitted, please tell and I'll revert.

By default, this PR additionally installs

• compiler-lib/*mli
• topdirs.mli
• arith_status.cmti big_int.cmti bigarray.cmti dynlink.cmti
  graphics.cmti graphicsX11.cmti nat.cmti num.cmti profiling.cmti
  ratio.cmti str.cmti topdirs.cmti unix.cmti unixLabels.cmti
• ocamldoc/odoc_info.cmti
• in threads: condition.cmti event.cmti mutex.cmti thread.cmti threadUnix.cmti
• in vmthreads: condition.cmti event.cmti mutex.cmti thread.cmti threadUnix.cmti
• ocamldoc/odoc_info.cmt
• profiling.cmt

The make target install-compiler-sources installs the ml files for compiler-libs.

@dbuenzli please check if this fixes MPR7362 completely.

@damiendoligez Please check.

@lefessan:

Every OCaml switch is already big (between 200 MB to 1 GB), which makes having a lot of them become very space consuming

If opam has a problem or if the way opam is used creates problems, then the people concerned about this should fix this in opam. But these opam problems should neither have a negative impact on people not using opam (and I believe there are quite a lot) nor stop us from taking the right decisions here.

What will happen if we double the space requirement by adding sources

Please argue with facts instead of dubious guesses. This PR adds 1.2MB of mli files and 4.5MB of ml files for the install-compiler-sources target. This is an increase of 0.6% and, respectively, 2.3% for the install-compiler-sources target. I really cannot see any point for creating fear about doubling space requirements.

I think a better approach would be for you to create an OPAM package ocaml-sources, with the same sources as the compiler,

IMHO, the solution would be that the opam maintainers create opam compiler packages that only install the essential stuff for those people that want to install all available OCaml versions in parallel.

If opam has a problem or if the way opam is used creates problems, then the people concerned about this should fix this in opam.

I don't think that's a problem specific to OPAM, system package manager might also want to have lean installs.

@dbuenzli please check if this fixes MPR7362 completely.

I guess so, though I don't know if maybe too much .cmti files are installed; only those that are part of the API should be installed --- but this is not for me to judge.

Regarding .cmt files I'm not sure there's an interest in installing those, maybe you want to connect their install to the *-compiler-source targets since they correspond to the .ml files.

The "apart from otherlibs" files seem perfectly fine to me, I'm quite sure if they were omitted it was by mistake (and in fact Topdirs is interesting to people playing to the toplevel).

The forthcoming gdb support for OCaml needs both .cmt files and source files to be accessible in order to work fully. One way of achieving this is using OPAMKEEPBUILDDIR=true, but that does use more disk space than is really required, since the .o and most of the other .cm* files are not required for debugging. (You also have to remember to actually set the variable...) Instead perhaps it is reasonable to consider putting the source files in the installation directory for compiler-libs. This needs some code changes so they can be located by the debugger, but that should be doable. For people writing code against compiler-libs this would probably improve usability.

@mshinwell It would be nice if you could make a summary on the platform mailing list about what you think packages should install by default (compilation flags and build artefacts).

The issue has been raised more than once without being conclusive.

For most topkg packages these changes could happen by simply rereleasing topkg without packages having to be re-released (for now topkg compiles in debug mode and installs the .mli and .cmti files for the modules that you indicate are part of your API).

@damiendoligez I rebased and addressed your comments, please check.

@dbuenzli I was not sure about the cmt files either, but make install installs almost all cmt files - I simply followed the pattern.

@mshinwell I am not sure I understand. Do you mean that the gdb support won't work properly, because the sources from many different directories are install flat in compiler-libs? If this is the case, I would suggest to wait until we actually see the problem. It is easy to copy the directory structure into the compiler-libs installation directory, but I would suggest to wait with this.

Thanks @hendriktews !