I pushed a simplified version of the PR (build the prefixed version in parallel to the unprefixed one). This way the existing build system is not touched at all which makes the change much easier to digest and we can trivially distribute both versions for backwards-compatibility. I have updated the PR description in consequence and removed the [POC] from the PR title since I am happy with this approach.

Taking a cue from #1569, I adapted the install target to install the prefixed ocamlcommon into +ocamlcommon, ocamlbytecomp into +ocamlbytecomp, etc. This way the prefixed and unprefixed version of compiler-libs can be installed simultaneously.

Some rough timings for make world.opt from a fresh checkout:

This PR:

real	7m37.383s
user	6m36.207s
sys	0m48.595s

trunk:

real	4m55.897s
user	4m12.304s
sys	0m34.882s

Thanks for working on this! I'm really looking forward not having to expunge toplevels as this leads to really confusing situations for users.

I wanted to give this a try in utop, which made me wonder how the new prefixed libraries should be presented to the user. Have you thought about this? The current prefixed libraries are available as compiler-libs.ocamlcommon, ..., should we make the new ones available as just ocaml_common, ocaml_bytecomp, ...?

BTW, if we compile the two versions in parallel, then we won't be able to easily share values between code using the unprefixed version and code using the prefixed version. This is not a huge deal, but we could avoid it generating the following implementations for modules of the unprefixed version:

[@@@deprecated "use Ocaml_common.X instead"]
include Ocaml_common.X

This should also cut build times.

If we keep the four top-level modules as above, it seems natural to also include an underscore in the archive/library names.

Maybe we should keep it as well for the sub-directory names?

Many thanks for this great piece of work, @nojb!

As you mentionned dependencies and just out of curiosity: according to
you, how much simpler would your task have been, were there no .depend files
to take into account?

Thanks for the comments!

I wanted to give this a try in utop, which made me wonder how the new prefixed libraries should be presented to the user. Have you thought about this? The current prefixed libraries are available as compiler-libs.ocamlcommon, ..., should we make the new ones available as just ocaml_common, ocaml_bytecomp, ...?

Yes, that's one possibility. Another one could be ocaml.common, ocaml.byteopt, etc.

BTW, if we compile the two versions in parallel, then we won't be able to easily share values between code using the unprefixed version and code using the prefixed version. This is not a huge deal, but we could avoid it generating the following implementations for modules of the unprefixed version:

[@@@deprecated "use Ocaml_common.X instead"]
include Ocaml_common.X

This should also cut build times.

This is a neat idea; I will give it a try.

If we keep the four top-level modules as above, it seems natural to also include an underscore in the archive/library names.

Maybe we should keep it as well for the sub-directory names?

Indeed.

As you mentionned dependencies and just out of curiosity: according to
you, how much simpler would your task have been, were there no .depend files
to take into account?

Dealing with dependencies took about 50% of the time so far, I would say.

Do you have something particular in mind?

I pushed one more commit implementing most of @diml's suggestion.

Differences with previous commit:

• The prefixed compiler-libs are built directly into compilerlibs (instead of prefixed_compilerlibs) to signal that they are the "main" artifact.

• Unprefixed files are generated in unprefixed_compilerlibs/XXX.{ml,mli}, e.g. unprefixed_compilerlibs/parsetree.mli containing

  include module of struct include Ocaml_common.Parsetree end

• The unprefixed compiler-libs are built out of these generated modules so they have types compatible with the prefixed ones, and compilation is much faster.

• The variables COMMON, BYTECOMP, etc in the main Makefile now contain lists of modules (no extension), and include even those which are "interface-only". Two new variables MLI_ONLY and ML_ONLY are used to keep track those modules which are interface- or implementation-only.

• The compiler itself (and most associated tools) are built using the unprefixed compiler-libs. Note that the unprefixed compiler-libs are not yet marked as deprecated.

Still left to do adapt:

• make install
• parsing/parser.ml (due to menhir --infer)
• ocamlmiddleend
• ocamlopttoplevel (these libraries have modules overlapping with the other compiler-libs, which requires some adjusting)
• ocamldoc
• debugger
• manual [disable generation of doc for compiler-libs for now - needs more support from ocamldoc]
• otherlibs/dynlink
• testsuite
• make check_all_arches

This will break all users of compiler-libs. To maintain backwards compatibility both the prefixed and unprefixed versions are distributed as a transitional measure.

How bad is that in practice ? Aren't most users using compiler-libs via the ocamlfind indirection ? If that is the case. Maybe only providing a new META file with the new names and tweaking the META file with the old names to point to the new ones with a deprecation warning predicate could be entirely sufficient no ?

• To get rid of the expunge process: this is necessary to allow several interesting improvements to the toplevel experience (see MPR#7589 and MPR#6704)

I'd to highlight that strictly speaking one does not need to get rid of the expunge process for improving the toplevel experience. Rather the proper functionality has to be exposed in the Toploop API and it should be made sure that this API does not leak expunged identifiers (more precisely exceptions) which is currently the case. I tend to think that it's maybe better to get rid of the expunge process but some people may think otherwise (e.g. depending on your viewpoint it puts more "garbage" in the initial environment).

I re-ran the script with the new revisions. I noticed that the cmt files for the unprefixed modules are no longer installed. Is that intentional? I would suggest to get rid of them at the same time as the unprefixed modules.

Thanks for the review. I pushed a fix for this along with several simplifications.

I also noticed that all the man pages for the compiler modules are gone. Personally, I'm fine with this change, but it might be worth mentioning it in the changelog.

Yes, they are disabled because ocamldoc cannot cope very well with prefixed libraries. @Octachron said he may have a look into adapting ocamldoc, but I would also not be shocked if we simply do not include docs for compiler-libs in the manual.

This may not be directly related to this PR, but at least ocamlcommon.cma could use plain -pack rather than this approach: it is linked with -linkall.
Is there anywhere an analysis of why this -linkall is needed?
It dramatically grows the size of executables using it, and prevents from linking with ocamlcommon.cma in the tools directory.

This may not be directly related to this PR, but at least ocamlcommon.cma could use plain -pack rather than this approach: it is linked with -linkall.
Is there anywhere an analysis of why this -linkall is needed?
It dramatically grows the size of executables using it, and prevents from linking with ocamlcommon.cma in the tools directory.

Good question, I am not sure; will try to do some git archaeology to see if it reveals anything.

This may not be directly related to this PR, but at least ocamlcommon.cma could use plain -pack rather than this approach: it is linked with -linkall.
Is there anywhere an analysis of why this -linkall is needed?
It dramatically grows the size of executables using it, and prevents from linking with ocamlcommon.cma in the tools directory.

The -linkall flag was added in #53

Recently (more or less) we added support for -linkall on individual compilation units (.cmo and .cmx files), not just on library files (.cma and .cmxa). So it would be possible to set -linkall on the compiler units that really need it because they self-initialize, instead of on the whole ocamlcommon.cm{,x}a library. Whether this would save significant code size is unclear, however.

Please check this on MacOS before you merge (for example with precheck) because Apple's version of gnumake is rather old and the Makefile changes are non-trivial.

I had sort of forgotten about this PR. For the record, I have been bitten by compiler-units-name conflicts related to compilerlibs recently (I had a lexer.mll file; the fix was to tell Dune to use (wrapped_executables true), so that the project's units were prefixed, avoiding the conflict), and I use JaneStreet's ocaml-compiler-libs package in outside projects using compiler-libs, so I already use say Ocaml_common.Lexer; this coding style is compatible with prefixing+aliases, and even if we don't do actually use prefixing in the compilerlibs today, I think we should make it possible for downstream users -- or clearly advertise ocaml-compiler-libs as the right middle-man.

From a high-level perspective, I think it's a royal pain that all OCaml projects have to consider unit prefixing/wrapping; Dune makes it somewhat hidden, but they remain ugly workarounds for the lack of a proper namespacing system -- or at least support mixing compilation units with the same file name and different digests in the same project (my own #2228 was a zero-th step in this direction, and I know @mshinwell and @lpw25 are working on this on-and-off as well).

I closed this PR because it was seriously out of date, but I am planning to take another shot at this shortly, following your #9211 which has rearranged some things in a useful way.