Generally I like the idea, and most of the changes proposed sound fine.

I don't have an opinion on the changes to float marshalling strategy. But: do we really need to marshal floats as part of the bootstrap? (I assume that this would come from float constants in the sources of boot/ocamlc, but do we really need those?) Could we instead forbid float constants when in_boot_compiler and fix the assumedly-small places where floats are currently used?

I don't think we can prohibit floats - for example, the constant 0.0 gets emitted as a result of this line in stdlib/camlinternalFormat.ml:

Just in case my description made it sound grander, the new flag to Marshal doesn't add any "new" code to extern, it just adds more control to the decisions taken with floats. extern already had the ability to "canonicalize" all floats as little-endian, it just by default doesn't do that on big-endian systems.

The other perfectly reasonable option would be either to actually prohibit bootstrapping on big-endian systems, just altering the first commit to reject the attempt or to accept that, a bit like regenerating configure with the wrong version of autoconf, if you happen to have an IBM mainframe (or a very old Power Mac G5) as your development machine, then you'll need to do your bootstrap commits somewhere else 🙂

Just for the sake of argument: why not always encode floats in little-endian mode? Are there any downsides to it (apart from making past marshalled floats incompatible with the new version)?

Just for the sake of argument: why not always encode floats in little-endian mode? Are there any downsides to it

Marshalling and unmarshalling of floats and esp. float arrays will be slower on big-endian machines, but:

• probably not by much;
• the only big-endian architecture that still matters for OCaml is Z Systems.

The current implementation is designed so that no endianness change is needed in the common case where the same architecture marshals then unmarshals FP numbers. Maybe it's too clever for its own good.

It also wouldn’t break incompatibility - there’d be no reason to remove the code for un-marshalling big-endian floats in intern.

Given that no one’s throwing hands up in horror at the initial parts (common config + stripping CRCS), shall I split this, so that this PR gives repeatable bootstrapping on non-big-endian systems and then move the marshal part to a separate PR? If Xavier (as extern’s main author) is happy for the simplification, I’d be happy to open the PR as being the complete removal of big-endian extern-ing. IIRC we either benchmarked or subsequently found some Marshal-sensitive workloads as part of the removal of the blue colour, so those could be reused to check how bad the impact is on Z?

Or, or course, just remove the Marshal stuff and block big-endian bootstrapping, but I don’t think anyone’s spoken up for that approach yet!

I think your proposed PR split makes a lot of sense.

Marshalling part removed, to be followed up separately.

Why does the bootstrap procedure need to be disabled when either flat-float-array or force-safe-string are disabled?

Two other questions:

Do we really want that boot/ocamlc has a way to know it is different
from another compiler? It makes me a bit nervous to make this compiler
"special".

Why did you have to change the SUBST mechanism? This complication
of the interface between the configuration and build system worries me a
bit, too.

Why does the bootstrap procedure need to be disabled when either flat-float-array or force-safe-string are disabled?

The disallowed options are those which would affect the generation of utils/config.ml (the safe-string is because I originally wrote this on the 4.x branch).

If you configure with --disable-flat-float-array then Config.flat_float_array should be false, but the bootstrap will ignore it and have Config.flat_float_array = true (otherwise utils/config.cmo would be different, and we lose the reproducibility).

Bootstrapping from a system which doesn't have flat float array doesn't fail per se, because the bytecode compiler doesn't containing any float array constants, but it does require a double-bootstrap (the first one will fail to reach a fix-point because of the configuration mismatch) - i.e. the first bootstrap systematically fails.

Two other questions:

Do we really want that boot/ocamlc has a way to know it is different from another compiler? It makes me a bit nervous to make this compiler "special".

We don't have to - it's used solely to display better error messages if the boot compiler were to be accidentally used to do something it must never do (today, the boot compiler must never be used to compile C files, but we nearly did that by mistake with caml-tex (remembering the conversation starting at #10366 (comment)). I agree it would be very worrying for that knowledge to be used for actual compilation/build decisions.

Originally, I added Config.in_boot_compiler to be sure for myself that the build system definitely wasn't using any of the values which I was setting to <boot compiler> but it also seemed worth keeping it in place to stop any risk of generating a command line beginning <boot compiler> -c foo.c etc.!

Why did you have to change the SUBST mechanism? This complication of the interface between the configuration and build system worries me a bit, too.

It doesn't have to be - the main thing is that I needed was a way of getting a consistent utils/config.ml. The alternative would be to have a static utils/config.boot.ml and select between that and the utils/Makefile-generated one? I don't mind - I think adapting SUBST was simply the solution which first occurred to me on the train when I wrote it last November :)

@shindere - I've pushed an alternate implementation in 4827373 which doesn't alter the generation in utils/Makefile. For that to work, I've done three things:

• The "code" which has ended up in utils/config.mlp is now in utils/config.common.ml
• The fixed values used for the boot compiler are statically declared in utils/config.boot.ml
• Both versions are built always in utils/config.boot.cmo and utils/config.main.cmo - this means that if you add a new constant to to config.mli and config.mlp but forget to add it to config.boot.ml then the very next build fails, rather than the next make bootstrap

It's slightly more invasive done that way, but equally the explicit nature of the boot compiler's configuration is also clearer.

We don't have to - it's used solely to display better error messages if the boot compiler were to be accidentally used to do something it must never do (today, the boot compiler must never be used to compile C files, but we nearly did that by mistake with caml-tex (remembering the conversation starting at #10366 (comment)). I agree it would be very worrying for that knowledge to be used for actual compilation/build decisions.

You're laying traps for the future, and you're peppering the compiler with patches that are only relevant to misuses of the bootstrap compiler. I don't think this is a good trade-off. IMO it would be better to do away with in_boot_compiler and fatal_in_boot_compiler and carefully design the configuration constants to make sure:

1. that they won't be executed as commands by the shell
2. that they contain a useful-enough error message

I've given an example above with c_compiler. I suggest getting rid of the '<' and '>' that could be misinterpreted by a shell and starting with / (i.e. shash-space) so that, even after tokenization, they can't be executable commands in any circumstance.
The error messages won't be displayed as nicely, but frankly I don't think it's worth adding tests to the compiler just to get nicer error messages when making the exotic mistake of compiling ordinary programs with the bootstrap compiler. That's not something ordinary users are ever likely to do.

Thanks for the review, @damiendoligez! I have reverted the peppering ("season to taste") - it was not for users, but for core devs... this error in CI (which with the sentinel Config.ext_dll = ".so_" now occurs everywhere rather than the more confusing Windows-only failure that actually happened in AppVeyor in #10366):

../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -g -use-prims ../runtime/primitives -I .. -nostdlib -I ../stdlib -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../middle_end -I ../middle_end/closure -I ../middle_end/flambda -I ../middle_end/flambda/base_types -I ../driver -I ../toplevel -I ../file_formats -I ../lambda -I ../otherlibs/str -I ../otherlibs/unix -linkall -o caml-tex -no-alias-deps ../compilerlibs/ocamlcommon.cma ../compilerlibs/ocamlbytecomp.cma ../compilerlibs/ocamltoplevel.cma ../otherlibs/str/str.cma ../otherlibs/unix/unix.cma caml_tex.ml
File "caml_tex.ml", line 1:
Error: I/O error: dllcamlstr.so_: No such file or directory

is less clear/explicit than this error:

../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -g -use-prims ../runtime/primitives -I .. -nostdlib -I ../stdlib -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../middle_end -I ../middle_end/closure -I ../middle_end/flambda -I ../middle_end/flambda/base_types -I ../driver -I ../toplevel -I ../file_formats -I ../lambda -I ../otherlibs/str -I ../otherlibs/unix -linkall -o caml-tex -no-alias-deps ../compilerlibs/ocamlcommon.cma ../compilerlibs/ocamlbytecomp.cma ../compilerlibs/ocamltoplevel.cma ../otherlibs/str/str.cma ../otherlibs/unix/unix.cma caml_tex.ml
>> Fatal error: Cannot load DLLs with the boot compiler
Fatal error: exception Misc.Fatal_error

12127c6 removes the special cases from cmpbyt and alters the end of the bootstrap cycle to strip the two artefacts which are in the tree. It's tempting (and cleaner) to get the rules for ocamlc$(EXE) and lex/ocamllex$(EXE) to do the stripping themselves, based on whether we're in the bootstrap cycle or not (given that everything has to be rebuilt at the end) but that gets fiddly quite quickly (in particular, tools/stripdebug isn't guaranteed to be useable at this point if you're in an actual bootstrap). At which point, it starts to be tempting to have options in ocamlc itself to do the stripping at which point this initially simple change balloons out of control......

The version I've done so far removes the logical duplication between tools/stripdebug.ml and tools/cmpbyt.ml and also makes the latter optional (because we could just use cmp), so it seems at least a partial win.

OK, last bit for this evening - this PR now makes no changes to cmpbyt. AppVeyor is revealing that I should have thought harder before trying to trick the compiler into compiling config.boot.ml with config.boot.mli. It was working in the other runs because config.mli is compiled just beforehand - the test for whether the foo.mli is compiled in Typemod uses the module, not the extension-stripped filename. I'll fix that tomorrow and then this should be green again...

The version just pushed should pass CI - I tidied up the recent commit history (the switch away from altering SUBST now starts at af06337)

Thanks for the review, @damiendoligez! I have reverted the peppering ("season to taste") - it was not for users, but for core devs... this error in CI (which with the sentinel Config.ext_dll = ".so_" now occurs everywhere rather than the more confusing Windows-only failure that actually happened in AppVeyor in #10366):

../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -g -use-prims ../runtime/primitives -I .. -nostdlib -I ../stdlib -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../middle_end -I ../middle_end/closure -I ../middle_end/flambda -I ../middle_end/flambda/base_types -I ../driver -I ../toplevel -I ../file_formats -I ../lambda -I ../otherlibs/str -I ../otherlibs/unix -linkall -o caml-tex -no-alias-deps ../compilerlibs/ocamlcommon.cma ../compilerlibs/ocamlbytecomp.cma ../compilerlibs/ocamltoplevel.cma ../otherlibs/str/str.cma ../otherlibs/unix/unix.cma caml_tex.ml
File "caml_tex.ml", line 1:
Error: I/O error: dllcamlstr.so_: No such file or directory

My suggestion was to set Config.ext_dll = ".so_The boostrap compiler cannot load DLLs" so you still get your nice error message displayed, admittedly in a slightly non-standard format.

Ah, that makes sense! I've updated the extensions with that and tidied the history so that there's only a single bootstrap. I'd quite like to keep the interim commits rather than squashing (as they encode different, but working, approaches), if that's OK.

The discussion in #11182 and the non-trivial bootstraps that entails provide a lovely testcase for automated repeatability, which I'd quite like to get on and add, therefore!

I've pushed a Changes entry - is there a consensus to merge this PR? Incidentally, I've described this as "repeatable" in the title, but that's actually the final objective - this PR of course makes the bootstrap reproducible - repeatability comes in the next PR where we test that each bootstrap artefact can be reproduced from the previous boot artefact.

I must say, this PR makes me a bit nervous in that it makes the build of the compiler a bit more complex than it already is, it feels. In particular, as mentionned earlier, the idea would be to get rid of all the SUBST machinery and have utils/config.ml produced by configure rather than by make.

This PR now no longer touches that at all - a future PR can start generating utils/config.main.ml in configure and then all that’s left is a cp for the main build and a cat of two static files for the bootstrap.

Is it really necessary to make the ubild more complex with regard to the expected result? Is it worth it?

I will be heretically honest - I don’t care about reproducible builds that much, but I do care about the second class contributor process we have where we have to have a core dev repeat bootstraps and manually merge them, because we can’t trust the process as it is now. So I think that the small bit of complexity to build boot/ocamlc with a fixed Config module is worth it, yes. Making this part of the build truly reproducible is an added bonus.

Thinking out loud: would it make sense to have a special configure mode to enable building the bootstrap compiler?

Bootstrapping is a development activity, though - you make changes on an already coldstart’d tree, bring the new system to stability and then use the coldstart’d compiler and the new one to do the bootstrap - reconfiguring to do that seems strange?

An alternate dance could be to have a way in ocamlc to link ocamlcommon.cma but override its Config module with a different one (gcc does this with .a files IIRC). However, adding new features to the compiler to facilitate this seems more complicated :)

Do you attempt to cleanup the history even more, or to leave it that way? Just asking.

The history's as I'd like it, yes :)

In .gitignore, the line /utils/config_*.ml* has been added twice.

Oops - I've deleted the duplicate on the last rebase.

Also, sorry if you already explained, but, why do --disable-flat-float-array and --disable-force-safe-string prevent from bootstrapping? Is it because that cwould introduce changes in the format of .cmi files and make the artifacts of the two compilers incompatible? If that's the case, isn't it true that any suchmodification which would create incompatibilities will also have to be tested?

I removed the safe-string part - there was nothing wrong with it, but it was left-over from when this was based on pre-5.x trunk. If you've configured with --disable-flat-float-array then the setting would contradict the let flat_float_array = true in utils/config.fixed.ml and you'd build a compiler which was lying about itself and which also would have a different binary representation, yes. Nothing needs testing, though. Any future configuration option which affects the bytecode compiler would just get added - the indicator that this needs doing is adding a new fixed value to utils/config.fixed.ml (in passing, flambda, afl and so forth do not matter because they don't affect the bytecode compiler).

Could the definition of IN_COREBOOT_CYCLE be moved to Makefile.build_config.in ?

It's not part of the build configuration, though, surely Makefile.common is the correct place? I expect virtually every line of the .in files to include things configure is substituting.

Also, I think we might already have discussed this in other places, but while reviewing hits PR I am wondering again. Why do we insist onmaking our Makefile rules depend on the configuration files and sometimes on the Makefiles themselves? To me this looks overkill and, I have to say, slightly insane.

The processing of those files directly depends on the recipe in utils/Makefile - if the files don't depend on it, then when working on it (as here), you have to touch the files every time to trigger rebuilds! I grant that having everything depend on the Makefiles would be pretty insane, but developing this PR itself would have been a pain without it (given the iterations)!

In utils/config.common.ml: As this is a new file, I think its header should refer to the persoon who created it, with proper affiliation (Tarides, I guess). "Portions of the Config module common to the both the boot and main": spurious "the". Same remark on the header of utils/config.fixed.ml.

There's nothing of mine in utils/config.common.ml - all I've done is move existing text around. Changing the header feels like plagiarism :) I've claimed the profound constants in utils/config.fixed.ml as mine :)