I am creating this issue to track and discuss a build failure that
occurs in the current trunk
when --enable-flambda --disable-flat-float-array are both set. I am
not sure what is the best way forward and hoping for a clarifying
discussion.

TL/DR: Questions

• Should we explicitly request that boot ocamlc and repo ocamlc
  produce the same .cmi files?

• If not, we have a dependency problem to fix. What do you think would
  be a good approach?

• In particular, would it be reasonable to stop using boot/ocamlc to
  build anything else that the ./ocamlc binary, and build the rest
  with ./ocamlc instead?

Naming:

• boot/ocamlc: boot compiler
• ./ocamlc: repo compiler

Context

The build failure only started after #2188 was merged, but my current
understanding is that it comes from a defect in the Makefile
dependencies that has presumably been there forever. We only hit
a build failure now because #2188 is such that,
when --disable-flat-float-array is set, the boot compiler
(compiled with --enable-flat-float-array) and the repo compiler
produce different .cmi files for modules of the standard library.

Problem

The current build process performs the following steps:

1. build the OCaml runtime and bytecode interpreter (ocamlrun)
2. use ocamlrun to build the standard library, using the boot compiler;
   copy stdlib artefacts into boot/ as the "boot stdlib"
3. use the boot compiler and the boot stdlib to build the repo compiler
4. use the repo compiler to build the stdlib again (the "repo stdlib")
5. use the boot compiler and boot stdlib to build the bytecode world
6. use the repo ocamlopt and the repo stdlib to build the native world

The dependency issues is as follows: at the beginning of step (6),
some compiler files that were compiled against the boot stdlib in step
(3) are linked with the repo stdlib built in step (4). This is correct
whenever the two stdlib have the same digests, which apparently was
always the case previously. When --disable-flat-float-array is set,
the bootstrap compiler and the repo compiler produce different digests
since #2188, which produces different type-declaration information
on --disable-flat-float-array.

Note: the reason why the stdlib is rebuilt (step 4) is that
stdlib/Makefile creates a dependency of stdlib artefacts over the
compiler currently used to build them (see COMPILER_DEPS in
stdlib/Makefile). In particular, when we request the "rest of the
build" after step (3), the repo compiler is newer than the stdlib,
so this dependency causes a rebuild.

We cannot fix the present problem with the same trick (by marking
a dependency of compiler files on the stdlib), because doing so
would create a dependency cycle: ocamlc depends on compiler files
that would depend on the stdlib which depends on ocamlc, triggering
and infinite rebuild whenever ocamlc changes.

No clear best path to a solution

I considered general several approaches to fix this, including the
following (many of them could turn out to be broken):

1. create a file indicating which phase of the compiler we are in
   (1-2-3-5 are "phase 1", 4-6 are "phase 2"), and having a dependency
   of compiler modules on this phase-file

2. remove the repo-compiler artifacts at the end of (5), to force
   a rebuild (removing artifacts during build sounds like a bad idea)

3. implement machinery to produce build artifacts in "build
   directories" instead of source directories, and use a separate build
   directory (boot/phase1?) for steps 1-2-3-5

These all look like a fair amount of work, so I thought that
discussing the general problem first would be a good first move.

Note: currently, boot/ocamlc remains used to build all core bytecode
targets, typically ocamlopt (in particular, note that ocaml{c,opt}
and ocaml{c,opt}.opt are built using different compiler eras, the
bytecode compilers are build using boot/ocamlc and the native
compilers with source ocamlopt).

If we wanted to get out of this general class of issues, I believe
that we should ensure that boot/ocamlc is only ever used to build
the ./ocamlc binary, and never afterwards. In particular, ocamlc
should be used for all followup operations. With that guarantee, we
could clean the non-boot/ build artifacts after ./ocamlc is
produced, and not worry about this problem.

Another way out?

There may be another way out, which is to require that the boot ocamlc
and repo ocamlc only ever produce the same .cmi files. If we had this
property (it generally holds in typical settings), we wouldn't need to
fix our dependency tracking.

For the concrete problem of #2188, we could restore this property by
adding a compile-time flag for {,no-}flat-float-array, for which the
configure-time flag would only select the default. Then we could run
the bootstrap compiler with the configured setting, and thus ensure
that boot/ocamlc and ./ocamlc produce exactly the same .cmi files.