Relocatable OCaml - to opam and beyond#14246
Conversation
shym
left a comment
There was a problem hiding this comment.
I read through this PR, but not testing it (yet). I like very much the cleanup in the installation rules!
I’ve taken a few notes, here and inlined.
Makefile
Outdated
|
|
||
| # These three targets are the slightly esoteric special sauce that avoid | ||
| # recursive make invocations in the install targets. | ||
| # There are three basic install recipies: |
There was a problem hiding this comment.
| # There are three basic install recipies: | |
| # There are three basic install recipes: |
Makefile
Outdated
| # native-install | ||
| # - The old installoptopt target is also available as full-installoptopt and | ||
| # installopt | ||
| # These sets of recipies are then welded together by these three dependency |
There was a problem hiding this comment.
| # These sets of recipies are then welded together by these three dependency | |
| # These sets of recipes are then welded together by these three dependency |
There was a problem hiding this comment.
Yeah, I appear to have a spelling blind spot there - consistently wrong, at least 🤷
| # - When configured with --disable-native-compiler, the install target simply | ||
| # depends on common-install (see above) |
There was a problem hiding this comment.
Would it make sense to deprecate explicitly the old targets?
There was a problem hiding this comment.
I was shying away from that, just because I expected that to happen ultimately as part of the final stages of the Makefile merging project
Makefile.common
Outdated
|
|
||
| INSTALL_ENSURE_DIR = \ | ||
| $(if $(filter undefined,$(origin DIR_CREATED_$(subst exec,,$(1))_$(2))),$\ | ||
| $(eval DIR_CREATED_$(1)_$(2):=)$\ |
There was a problem hiding this comment.
| $(eval DIR_CREATED_$(1)_$(2):=)$\ | |
| $(eval DIR_CREATED_$(subst exec,,$(1))_$(2):=)$\ |
IIUC
There was a problem hiding this comment.
Yes, indeed, thanks - there should presumably be a stray mkdir in the install instructions
| $(SH_AND) | ||
|
|
||
| MK_LINK = \ | ||
| (cd "$(INSTALL_SECTION_$(2))$(addprefix /,$(3))" && \ |
There was a problem hiding this comment.
Why not use QUOTE_SINGLE here also?
There was a problem hiding this comment.
Hmm, it's not necessarily the most acceptable of explanations, but my reasoning was only to be using QUOTE_SINGLE where I can't be certain that single quotes won't appear (I may not have been consistent, though, given that the two arguments being linked are being passed to QUOTE_SINGLE). The rationale is that cmd makes all of this a bit of a nightmare anyway... i.e. the build system itself should never introduce something as a stupid as an apostrophe... but we must cope with the fact that a user path may quite legitimately include one! (I mean that while I wouldn't, I think it should be possible to install to C:\David's Work, but I think we would be very silly to engineer ocamlopt's finest binary.exe as one of our filenames!)
Makefile.common
Outdated
| # $2 = subdirectory (may be empty, but otherwise must end with "/") | ||
| # $3 = name to install $1 (must be specified) | ||
| # The compiler is installed as the ocaml package in opam, but the actual files | ||
| # are installed from other packages (typically ocaml-compiler). For the the lib |
There was a problem hiding this comment.
| # are installed from other packages (typically ocaml-compiler). For the the lib | |
| # are installed from other packages (typically ocaml-compiler). For the lib |
tools/opam/generate.ml
Outdated
|
|
||
| let output_endline oc = Printf.kfprintf (fun oc -> output_char oc '\n') oc | ||
|
|
||
| (* [generate_file file] processes then erases opam-bin, opam-lib opam-libexec |
There was a problem hiding this comment.
| (* [generate_file file] processes then erases opam-bin, opam-lib opam-libexec | |
| (* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec |
I suppose
| let to_backslashes oc s = | ||
| output_string oc (String.concat {|\\|} (String.split_on_char '/' s)) | ||
| in | ||
| output_endline oc | ||
| {| cmd /c "mklink %a\\%s %s"|} to_backslashes dir source target |
There was a problem hiding this comment.
No Filename.quote is needed there? (That might be harder, because it will end up in string, sigh)
There was a problem hiding this comment.
The quoting here is absolutely hellish - the problem is that mklink is not an executable. I went on the basis that the user who decides to do ./configure --host=x86_64-w64-mingw32 --bindir 'C:\OCaml\I like bin's\' can flippin' well contribute the patch to quote it 😂
.github/workflows/build-msvc.yml
Outdated
| opam init --bare --yes --disable-sandboxing --auto-setup --cygwin-local-install | ||
| opam switch create '${{ env.OPAMSWITCH }}' --empty | ||
| opam pin add --no-action --kind=path ocaml-variants . | ||
| opam pin add --no-action flexdll flexdll | ||
| opam pin add --no-action winpthreads winpthreads | ||
| opam install --yes flexdll winpthreads | ||
| opam install --yes --assume-built ocaml-variants | ||
| opam exec -- ocamlc -v |
There was a problem hiding this comment.
I suppose that the opam cli is quite stable at this point; but we could pass --cli=2.4 or set OPAMCLI for future-proofing.
There was a problem hiding this comment.
I should indeed use the features of opam that I originally designed/specified and recommend others use!! 🫣😂
| cd "%APPVEYOR_BUILD_FOLDER%" | ||
| appveyor DownloadFile "https://github.com/ocaml/flexdll/archive/%FLEXDLL_VERSION%.tar.gz" -FileName "flexdll.tar.gz" || exit /b 1 | ||
| appveyor DownloadFile "https://github.com/ocaml/flexdll/releases/download/%FLEXDLL_VERSION%/flexdll-bin-%FLEXDLL_VERSION%.zip" -FileName "flexdll.zip" || exit /b 1 | ||
| appveyor DownloadFile "https://github.com/ocaml/opam/releases/download/2.4.1/opam-2.4.1-x86_64-windows.exe" -FileName "opam.exe" || exit /b 1 |
There was a problem hiding this comment.
You could introduce OPAM_VERSION somewhere.
There was a problem hiding this comment.
To what end, though - it's used exactly once here? I'm never sure about polluting the environment in these horrible batch files - FLEXDLL_VERSION has to be communicated between three different files, but this one's localised.
shym
left a comment
There was a problem hiding this comment.
I tested the installation of a cross compiler, using both the standard and the opam modes and it worked as hoped, congratulations!
I could trim down a little bit the dummy files that are created just to reuse the common installation workflow:
diff --git i/Makefile.cross w/Makefile.cross
index b0a21c3d7c..362eb15aa9 100644
--- i/Makefile.cross
+++ w/Makefile.cross
@@ -105,11 +105,7 @@ INSTALL_OVERRIDES=build_ocamldoc=false WITH_DEBUGGER= OCAMLRUN=ocamlrun
.PHONY: installcross
installcross:
# Create dummy files to keep `install` happy
- touch \
- $(addprefix toplevel/, \
- $(foreach ext,cmi cmt cmti cmx, native/nat__dummy__.$(ext)) \
- all__dummy__.cmx topstart.o native/tophooks.cmi)
+ touch toplevel/topstart.o
$(LN) `command -v ocamllex` lex/ocamllex.opt$(EXE)
- $(LN) `command -v ocamlyacc` yacc/ocamlyacc.opt$(EXE)
# Real installation
$(MAKE) install $(INSTALL_OVERRIDES)
Oo, it's intentional (they should never have been being installed in the first place), but I'm not sure I should be doing it so subtly, indeed! It was a misunderstanding that crept in with the tsan changes, I think - we never need stub DLLs for native code (because .cmxs is already a stub DLL...)
Yes - I should be explicit about explicitness, indeed, given that all the paths are relative!! Thank you 🙂 |
I thought it strange for them to be dropped without a mention in the commit message, even if I didn’t know when (that is to say, even just if) they were useful. Maybe we could drop the |
3aeff31 to
4eb9eec
Compare
|
Rebased - review responses to follow |
4eb9eec to
c2acd9e
Compare
c2acd9e to
ad7e0ca
Compare
shym
left a comment
There was a problem hiding this comment.
I went through the range-diff and the “Review” commits (thank you so much yet again for such a clean update!): I managed to find only a missing comma :-D So this looks good to me!
| (* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec | ||
| and opam-man to produce [file] *) |
There was a problem hiding this comment.
| (* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec | |
| and opam-man to produce [file] *) | |
| (* [generate_install file] processes then erases opam-bin, opam-lib, | |
| opam-libexec and opam-man to produce [file] *) |
damiendoligez
left a comment
There was a problem hiding this comment.
Approved after interactive review; still some final testing to do.
937f9f6 to
3c2f8c8
Compare
c47392a to
dc9a4fd
Compare
All of the .mli files in toplevel/byte are "common", so they're already installed by the patterns in toplevel.
It's necessary to install main.o and optmain.o because the two modules are not part of ocamlbytecomp.cmxa and ocamloptcomp.cmxa, but the .cmx files are already installed as part of wildcard patterns on driver/
The ocamldoc installation commands in installopt duplicate a series of commands which already happen in install.
The ocamldoc binaries were installed using a relative path. They're the only binaries installed that way - switch them to use an implicit path, as it's easier to make an implicit path subsequently explicitly relative if needed than vice versa.
make install works as it did before. All the commands in the install targets now go through a macro call which allows the semantic intent of each command to be more clearly specified.
make [INSTALL_MODE=install] install - installs the compiler, as normal make INSTALL_MODE=display install - displays the operations needed for make INSTALL_MODE=list install - lists the files and symbolic links which are installed
976bb76 to
76ef1bc
Compare
tools/opam/process.sh
Outdated
| #* * | ||
| #************************************************************************** | ||
|
|
||
| set -e |
There was a problem hiding this comment.
for some extra safety maybe?
| set -e | |
| set -eu | |
| if set -o pipefail > /dev/null 2> /dev/null; then | |
| # POSIX.1-2024 | |
| set -o pipefail | |
| fi |
There was a problem hiding this comment.
same question for the generated -clone and -fixup scripts
There was a problem hiding this comment.
Yes, indeed - that's a good idea!
There was a problem hiding this comment.
What am I missing in calling it twice? Isn't
set -o pipefail > /dev/null 2> /dev/null || truesufficient?
There was a problem hiding this comment.
Hah - it turns out neither of us appears to have been correct! Apparently shell builtins are allowed to abort execution unconditionally, so it appears to be necessary to have:
if (set -o pipefail 2>/dev/null); then
set -o pipefail
fiWith Ubuntu's default dash, both this:
#!/bin/sh
set -o pipefail || echo No pipefailand
#!/bin/sh
if ! set -o pipefail; then
echo No pipefail
fidisplay an error from the set command but then immediately abort the script.
kit-ty-kate
left a comment
There was a problem hiding this comment.
I only took a couple of hours to briefly look at tools/opam/* but the general idea looks good, albeit a bit complex to decipher at first between all the various scripts and the planned opam-repository changes. I only had a couple of minor remarks that can be ignored if i was off (the opam --safe/readonly bits should probably be added though).
Hopefully opam will be able to clone relocatable packages itself in the future, which should allow to shave off a bit of the complexity by removing the clone mode, but that will probably need some time to be designed and implemented correctly.
Also the addition of the .install file is a really nice improvement i think.
tools/opam/process.sh
Outdated
| #* * | ||
| #************************************************************************** | ||
|
|
||
| set -e |
There was a problem hiding this comment.
same question for the generated -clone and -fixup scripts
| let _ = | ||
| let create_dir seen (dir, _, _) = | ||
| if not (StringSet.mem dir seen) && String.contains dir '/' then | ||
| output_endline oc {|mkdir -p '%s'|} dir; |
There was a problem hiding this comment.
It's probably best to avoid the silent invariant that no ' character exists in dir
| output_endline oc {|mkdir -p '%s'|} dir; | |
| output_endline oc {|mkdir -p %s|} (Filename.quote dir); |
There was a problem hiding this comment.
That just replaces it with a different invariant on Windows (Filename.quote doesn't do the same thing). I think it's more hassle than it's worth, given that this structure is only the root directories of opam switches (lib/ocaml, bin, etc.)
tools/opam/generate.ml
Outdated
| String.map (function '@' -> '/' | c -> c) | ||
| (String.sub file 6 (String.length file - 6)) | ||
| in | ||
| output_endline oc {|mkdir -p "$1/%s"|} dir; |
There was a problem hiding this comment.
| output_endline oc {|mkdir -p "$1/%s"|} dir; | |
| output_endline oc {|mkdir -p "$1"/%s|} (Filename.quote dir); |
tools/opam/generate.ml
Outdated
| In_channel.fold_lines (fun _ line -> | ||
| match String.split_on_char ' ' line with | ||
| | [source; dest] -> | ||
| output_endline oc {|cp '%s' "$1/%s/%s"|} source dir dest |
There was a problem hiding this comment.
| output_endline oc {|cp '%s' "$1/%s/%s"|} source dir dest | |
| output_endline oc {|cp %s "$1"/%s/%s|} | |
| (Filename.quote source) (Filename.quote dir) (Filename.quote dest) |
tools/opam/generate.ml
Outdated
|
|
||
| let clone_files oc dir ic = | ||
| output_endline oc | ||
| {|dest="$1/%s" xargs sh "$1/clone-files" <<'EOF'|} dir; |
There was a problem hiding this comment.
| {|dest="$1/%s" xargs sh "$1/clone-files" <<'EOF'|} dir; | |
| {|dest="$1"/%s xargs sh "$1/clone-files" <<'EOF'|} (Filename.quote dir); |
|
I like the |
76ef1bc to
f416c48
Compare
|
Thanks, @kit-ty-kate! I haven't changed the |
|
Separately, this has been tortured on precheck#1091 with a special check that builds the compiler and the trunk commit it sits on and compares the installation artefacts (i.e. checks that the same files are installed) and then repeats the entire process for |
91d84b9 to
b7b3efd
Compare
make INSTALL_MODE=opam install generates $OPAM_PACKAGE_NAME.install and $OPAM_PACKAGE_NAME-fixup.sh ($OPAM_PACKAGE_NAME defaults to ocaml-compiler). Nothing is installed by this mode. The fixup.sh script is intentionally not made executable (it should be invoked explicitly with sh) and creates symbolic links, if required, and also manually copies the files to the doc dir, as the .install file format doesn't allow the correct location to be specified.
Redundant filter, but mitigates an issue with --assume-built in opam
Mitigates an issue with opam install --assume-built since build dependencies are ignored. The semantics should remain consistent: in particular, as there is only a single version of each of these packages, the key issue is that removing the package will still trigger the correct behaviour as the dependency graph will change.
b7b3efd to
f13b75e
Compare
|
Updated torture version going through precheck#1092. I've added a commit at the end of the series which adds guards in the Makefile-machinery to detect filenames which would cause problem, I've also - for the sake of history, and because it was a useful extra check - ensured that the shell and OCaml generators are actually producing the same scripts and hardened the use of single-quoting on all-generated filenames. |
Makefile.common
Outdated
| # $(1) can be a mix of filenames and globs. Check $(1) itself for illegal | ||
| # characters (i.e. ensure that the whole of $(1) is a "valid" path) and then for | ||
| # each word containing a pattern character also evaluate the pattern and check | ||
| # the resulting files. $(2)/$(3) be be a valid section. |
f13b75e to
114231d
Compare
|
Last couple of tweaking notes to those macros added. However, @shym and I discussed this separately - the benefit of the make macros is that it would catch errors just doing a normal |
Additional hardening on the requirements for names of files and directories in the distribution.
114231d to
7cf5ef2
Compare
|
Assuming everything passes CI again here, and everyone remains happy, I think this is good to go, too. Relocatable and now easy to clone, too...! |
|
This happened on #14244 as well - the AppVeyor issue is that the "CI: Full matrix" check is occasionally pushing this over its 1 hour cancellation mark on the occasions when we get an over-stretched worker. |
This fourth PR of the trilogy forms the epilogue of Relocatable OCaml, providing the support to allow opam to capitalise on the new features to be able to clone an existing compiler whose configuration matches that requested by the user for a new opam switch. Note that this PR is entirely independent of the other three (in particular, this epilogue can in fact be merged before the other three PRs).
Fundamentally, this changes the various
installtargets to consist of a series of macro calls instead of commands. In normal use, these translate to the same commands as before (i.e.make installstill does the same thing), but the meta-programming of these targets then allows alternative installation mechanisms to be fairly trivially implemented.Commit-by-commit, to begin with:
installrecipe forocamldocuses the$(OCAMLDOC)and$(OCAMLDOC_OPT)variables to refer to the binaries. These are also used as part of the mechanism for selecting the "best"ocamldoc(so that the generation of manpages and so forth usesocamldoc.optif available) but this means that the name is prefixed with$(ROOTDIR). The effect is that instead of installingocamldoc/ocamldoc[.exe]as we do for all other files, we install./ocamldoc/ocamldoc[.exe]. The./creates problems in an opam.install, which expects implicit paths only.installcommand. In order to indirect this via a macro call, shellforloops can't be used, so there's a bit of necessary GNU make glue to allow the install recipe to be written in bits (install::; the same aspartialclean) and then theforloops which were used to install the various binaries intoolscan be converted to a$(foreach ...)which expands to explicit calls. It's noisy, but a mostly mechanical change...ifblocks are similarly hairy, and it is somewhat easier to reason about the branching if the macros are responsible for all the logic and the shell simply runs commands. The concern here is that a make macro executes, making a persistent decision (for example to create a directory) which ends up nested in a shellifwhose conditional is not activated.COMPILER_ARTEFACT_DIRSmacro tries to reduce the size of them a little bit, and make it slightly easier to maintain consistency (cf. the typo which had existed, albeit in a rarely used target...)INSTALL_STRIPPED_BYTE_PROGgets tweaked slightly not to delete the file each time (it still always creates it) so that a.installfile can refer to itSUBDIR_NAMEto go withROOTDIR- i.e.$(ROOTDIR)/$(SUBDIR_NAME)Finally, with everything somewhat more uniform, the commands used for installation are replaced with one of two macro calls. Each macro receives the section it installs to and optionally a subdirectory within that section. The four sections
bin,doc,libandmancorrespond to--bindir,--docdir,--libdirand--mandiras passed toconfigure.libexecalso installs to--libdir, but marks the file as executable (.sofiles,expunge, etc.). Finally,stublibsis an "alias" for thestublibssubdirectory, sinceopamhas a specific installation section for these.mkdir '$(LIBDIR)/compiler-libs$'is no longer necessaryinstall *.cmi '$(LIBDIR)/compiler-libs'becomes$(call INSTALL_ITEMS, *.cmi, lib, compiler-libs)INSTALL_ITEMwhich takes a single source file, a destination directory. The file can be installed with a different name and a list of basenames for symlinks in the same directory can be specified. For example, the installationtools/ocamlobjinfoin a bytecode-only build becomes$(call INSTALL_ITEM, tools/ocamlobjinfo, bin, , ocamlobjinfo.byte, ocamlobjinfo)which installs the executable itself asocamlobjinfo.byteand then creates a symlinkocamlobjinfolinking to it.INSTALL_BEGINandINSTALL_ENDare called, unsurprisingly, at the beginning and end of theinstallrecipe and they are called exactly once.The macro
INSTALL_MODEcontrols howmake installwill behave. It defaults toinstall. For each supported valuefooforINSTALL_MODE, there is then a set of macrosINSTALL_DESPATCH_foo_MKDIR,INSTALL_DESPATCH_foo_ITEM, etc., for the 4 installation macros. The macro machinery forINSTALL_ITEM, etc., allow a slightly less unnatural use of spaces (in GNU make,$(call FOO, a)and$(call FOO,a)are not the same call) and also for arguments to be omitted without triggering undefined variable warnings._DESPATCH_macros are then able to written slightly less unnaturally.5 installation modes are implemented:
install(the default) callsmkdir,installandln/cp, as beforedisplayis a "dry-run" version ofinstalllistlists each target file in its installed location (expanding wildcards). opam install files cannot contain duplicate targets, so this mode was useful in identifying some of the earlier de-duplication commitsopamuses theOPAM_PACKAGE_NAMEvariable (which defaults toocaml-compiler) and writes$(OPAM_PACKAGE_NAME).installand$(OPAM_PACKAGE_NAME)-fixup.sh. The shell script deals with two shortcomings (from the compiler's perspective) of opam install files. Thedocsection in the install file would install todoc/ocaml-compiler, rather thandoc/ocamland there currently no way to override this. The install format also cannot copy or create symlinks.ocaml-variants.opamin the repo is updated to use this mechanism, but I don't suggest we deploy it to the release packages until a future version of opam deals with these two limitations (or we do something which means they're no longer an issue)clonewrite a single$(OPAM_PACKAGE_NAME)-clone.shscript. This shell script is designed to be run from the installation prefix and clones the compiler to the single argument argument it's passed. This is the key mechanism needed for Relocatable OCaml, as installing this script into opam switches with the compiler allows that compiler to be cloned to another switch.The recommended workflow for installing a built compiler was inadvertently broken in #13160. The underlying problem is tracked in ocaml/opam#6621 but two commits which workaround this are added which then allows the compiler built in GitHub Actions to be tested as an opam switch (in addition to providing CI for this somewhat advanced workflow, it also means that the opam file is tested in CI without having to build the compiler again).