Skip to content

Relocatable OCaml - to opam and beyond#14246

Merged
NickBarnes merged 23 commits intoocaml:trunkfrom
dra27:opam-install-file
Dec 17, 2025
Merged

Relocatable OCaml - to opam and beyond#14246
NickBarnes merged 23 commits intoocaml:trunkfrom
dra27:opam-install-file

Conversation

@dra27
Copy link
Copy Markdown
Member

@dra27 dra27 commented Sep 15, 2025

This fourth PR of the trilogy forms the epilogue of Relocatable OCaml, providing the support to allow opam to capitalise on the new features to be able to clone an existing compiler whose configuration matches that requested by the user for a new opam switch. Note that this PR is entirely independent of the other three (in particular, this epilogue can in fact be merged before the other three PRs).

Fundamentally, this changes the various install targets to consist of a series of macro calls instead of commands. In normal use, these translate to the same commands as before (i.e. make install still does the same thing), but the meta-programming of these targets then allows alternative installation mechanisms to be fairly trivially implemented.

Commit-by-commit, to begin with:

  • The first four commits deal with various instances where we currently files more than once. While only trivial wasteful at the moment, it's problematic for generating opam files, where a destination file can only be specified once.
  • The install recipe for ocamldoc uses the $(OCAMLDOC) and $(OCAMLDOC_OPT) variables to refer to the binaries. These are also used as part of the mechanism for selecting the "best" ocamldoc (so that the generation of manpages and so forth uses ocamldoc.opt if available) but this means that the name is prefixed with $(ROOTDIR). The effect is that instead of installing ocamldoc/ocamldoc[.exe] as we do for all other files, we install ./ocamldoc/ocamldoc[.exe]. The ./ creates problems in an opam .install, which expects implicit paths only.
  • Currently, files are installed simply by invoking the install command. In order to indirect this via a macro call, shell for loops can't be used, so there's a bit of necessary GNU make glue to allow the install recipe to be written in bits (install::; the same as partialclean) and then the for loops which were used to install the various binaries in tools can be converted to a $(foreach ...) which expands to explicit calls. It's noisy, but a mostly mechanical change...
  • In the same vein, shell if blocks are similarly hairy, and it is somewhat easier to reason about the branching if the macros are responsible for all the logic and the shell simply runs commands. The concern here is that a make macro executes, making a persistent decision (for example to create a directory) which ends up nested in a shell if whose conditional is not activated.
  • We have patterns everywhere for the installation of artefacts - the COMPILER_ARTEFACT_DIRS macro tries to reduce the size of them a little bit, and make it slightly easier to maintain consistency (cf. the typo which had existed, albeit in a rarely used target...)
  • INSTALL_STRIPPED_BYTE_PROG gets tweaked slightly not to delete the file each time (it still always creates it) so that a .install file can refer to it
  • The "Straighten out the..." commit consolidates the various variables used to represent installation names (I expect this was already planned once the last few remaining Makefiles were merged). In the same vein, the unmerged Makefiles which also install things now have SUBDIR_NAME to go with ROOTDIR - i.e. $(ROOTDIR)/$(SUBDIR_NAME)

Finally, with everything somewhat more uniform, the commands used for installation are replaced with one of two macro calls. Each macro receives the section it installs to and optionally a subdirectory within that section. The four sections bin, doc, lib and man correspond to --bindir, --docdir, --libdir and --mandir as passed to configure. libexec also installs to --libdir, but marks the file as executable (.so files, expunge, etc.). Finally, stublibs is an "alias" for the stublibs subdirectory, since opam has a specific installation section for these.

  • mkdir '$(LIBDIR)/compiler-libs$' is no longer necessary
  • install *.cmi '$(LIBDIR)/compiler-libs' becomes $(call INSTALL_ITEMS, *.cmi, lib, compiler-libs)
  • Items which need renaming and for which symlinks are required use INSTALL_ITEM which takes a single source file, a destination directory. The file can be installed with a different name and a list of basenames for symlinks in the same directory can be specified. For example, the installation tools/ocamlobjinfo in a bytecode-only build becomes $(call INSTALL_ITEM, tools/ocamlobjinfo, bin, , ocamlobjinfo.byte, ocamlobjinfo) which installs the executable itself as ocamlobjinfo.byte and then creates a symlink ocamlobjinfo linking to it.
  • INSTALL_BEGIN and INSTALL_END are called, unsurprisingly, at the beginning and end of the install recipe and they are called exactly once.

The macro INSTALL_MODE controls how make install will behave. It defaults to install. For each supported value foo for INSTALL_MODE, there is then a set of macros INSTALL_DESPATCH_foo_MKDIR, INSTALL_DESPATCH_foo_ITEM, etc., for the 4 installation macros. The macro machinery for INSTALL_ITEM, etc., allow a slightly less unnatural use of spaces (in GNU make, $(call FOO, a) and $(call FOO,a) are not the same call) and also for arguments to be omitted without triggering undefined variable warnings. _DESPATCH_ macros are then able to written slightly less unnaturally.

5 installation modes are implemented:

  • install (the default) calls mkdir, install and ln/cp, as before
  • display is a "dry-run" version of install
  • list lists each target file in its installed location (expanding wildcards). opam install files cannot contain duplicate targets, so this mode was useful in identifying some of the earlier de-duplication commits
  • opam uses the OPAM_PACKAGE_NAME variable (which defaults to ocaml-compiler) and writes $(OPAM_PACKAGE_NAME).install and $(OPAM_PACKAGE_NAME)-fixup.sh. The shell script deals with two shortcomings (from the compiler's perspective) of opam install files. The doc section in the install file would install to doc/ocaml-compiler, rather than doc/ocaml and there currently no way to override this. The install format also cannot copy or create symlinks. ocaml-variants.opam in the repo is updated to use this mechanism, but I don't suggest we deploy it to the release packages until a future version of opam deals with these two limitations (or we do something which means they're no longer an issue)
  • clone write a single $(OPAM_PACKAGE_NAME)-clone.sh script. This shell script is designed to be run from the installation prefix and clones the compiler to the single argument argument it's passed. This is the key mechanism needed for Relocatable OCaml, as installing this script into opam switches with the compiler allows that compiler to be cloned to another switch.

The recommended workflow for installing a built compiler was inadvertently broken in #13160. The underlying problem is tracked in ocaml/opam#6621 but two commits which workaround this are added which then allows the compiler built in GitHub Actions to be tested as an opam switch (in addition to providing CI for this somewhat advanced workflow, it also means that the opam file is tested in CI without having to build the compiler again).

@dra27 dra27 added no-change-entry-needed run-crosscompiler-tests Makes the CI run the Cross compilers test workflow CI: Full matrix Makes the CI test a bigger set of configurations labels Sep 15, 2025
@dra27 dra27 added the relocatable towards a relocatable compiler label Sep 15, 2025
Copy link
Copy Markdown
Contributor

@shym shym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through this PR, but not testing it (yet). I like very much the cleanup in the installation rules!
I’ve taken a few notes, here and inlined.

  • In a0933f2, installation of native DLLs is removed: was that on purpose?
  • In 79c7fea commit message, “relative” means “explicit”, I guess?
  • 515e861 creates a very elegant solution, congratulations :-)
  • 7fee32c: what a relief! :-)

Makefile Outdated

# These three targets are the slightly esoteric special sauce that avoid
# recursive make invocations in the install targets.
# There are three basic install recipies:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# There are three basic install recipies:
# There are three basic install recipes:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥧

Makefile Outdated
# native-install
# - The old installoptopt target is also available as full-installoptopt and
# installopt
# These sets of recipies are then welded together by these three dependency
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# These sets of recipies are then welded together by these three dependency
# These sets of recipes are then welded together by these three dependency

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I appear to have a spelling blind spot there - consistently wrong, at least 🤷

Comment on lines +2729 to +2730
# - When configured with --disable-native-compiler, the install target simply
# depends on common-install (see above)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to deprecate explicitly the old targets?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was shying away from that, just because I expected that to happen ultimately as part of the final stages of the Makefile merging project

Makefile.common Outdated

INSTALL_ENSURE_DIR = \
$(if $(filter undefined,$(origin DIR_CREATED_$(subst exec,,$(1))_$(2))),$\
$(eval DIR_CREATED_$(1)_$(2):=)$\
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$(eval DIR_CREATED_$(1)_$(2):=)$\
$(eval DIR_CREATED_$(subst exec,,$(1))_$(2):=)$\

IIUC

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, indeed, thanks - there should presumably be a stray mkdir in the install instructions

$(SH_AND)

MK_LINK = \
(cd "$(INSTALL_SECTION_$(2))$(addprefix /,$(3))" && \
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use QUOTE_SINGLE here also?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it's not necessarily the most acceptable of explanations, but my reasoning was only to be using QUOTE_SINGLE where I can't be certain that single quotes won't appear (I may not have been consistent, though, given that the two arguments being linked are being passed to QUOTE_SINGLE). The rationale is that cmd makes all of this a bit of a nightmare anyway... i.e. the build system itself should never introduce something as a stupid as an apostrophe... but we must cope with the fact that a user path may quite legitimately include one! (I mean that while I wouldn't, I think it should be possible to install to C:\David's Work, but I think we would be very silly to engineer ocamlopt's finest binary.exe as one of our filenames!)

Makefile.common Outdated
# $2 = subdirectory (may be empty, but otherwise must end with "/")
# $3 = name to install $1 (must be specified)
# The compiler is installed as the ocaml package in opam, but the actual files
# are installed from other packages (typically ocaml-compiler). For the the lib
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# are installed from other packages (typically ocaml-compiler). For the the lib
# are installed from other packages (typically ocaml-compiler). For the lib


let output_endline oc = Printf.kfprintf (fun oc -> output_char oc '\n') oc

(* [generate_file file] processes then erases opam-bin, opam-lib opam-libexec
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(* [generate_file file] processes then erases opam-bin, opam-lib opam-libexec
(* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec

I suppose

Comment on lines +121 to +125
let to_backslashes oc s =
output_string oc (String.concat {|\\|} (String.split_on_char '/' s))
in
output_endline oc
{| cmd /c "mklink %a\\%s %s"|} to_backslashes dir source target
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Filename.quote is needed there? (That might be harder, because it will end up in string, sigh)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quoting here is absolutely hellish - the problem is that mklink is not an executable. I went on the basis that the user who decides to do ./configure --host=x86_64-w64-mingw32 --bindir 'C:\OCaml\I like bin's\' can flippin' well contribute the patch to quote it 😂

Comment on lines +214 to +221
opam init --bare --yes --disable-sandboxing --auto-setup --cygwin-local-install
opam switch create '${{ env.OPAMSWITCH }}' --empty
opam pin add --no-action --kind=path ocaml-variants .
opam pin add --no-action flexdll flexdll
opam pin add --no-action winpthreads winpthreads
opam install --yes flexdll winpthreads
opam install --yes --assume-built ocaml-variants
opam exec -- ocamlc -v
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose that the opam cli is quite stable at this point; but we could pass --cli=2.4 or set OPAMCLI for future-proofing.

Copy link
Copy Markdown
Member Author

@dra27 dra27 Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should indeed use the features of opam that I originally designed/specified and recommend others use!! 🫣😂

cd "%APPVEYOR_BUILD_FOLDER%"
appveyor DownloadFile "https://github.com/ocaml/flexdll/archive/%FLEXDLL_VERSION%.tar.gz" -FileName "flexdll.tar.gz" || exit /b 1
appveyor DownloadFile "https://github.com/ocaml/flexdll/releases/download/%FLEXDLL_VERSION%/flexdll-bin-%FLEXDLL_VERSION%.zip" -FileName "flexdll.zip" || exit /b 1
appveyor DownloadFile "https://github.com/ocaml/opam/releases/download/2.4.1/opam-2.4.1-x86_64-windows.exe" -FileName "opam.exe" || exit /b 1
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could introduce OPAM_VERSION somewhere.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To what end, though - it's used exactly once here? I'm never sure about polluting the environment in these horrible batch files - FLEXDLL_VERSION has to be communicated between three different files, but this one's localised.

Copy link
Copy Markdown
Contributor

@shym shym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the installation of a cross compiler, using both the standard and the opam modes and it worked as hoped, congratulations!
I could trim down a little bit the dummy files that are created just to reuse the common installation workflow:

diff --git i/Makefile.cross w/Makefile.cross
index b0a21c3d7c..362eb15aa9 100644
--- i/Makefile.cross
+++ w/Makefile.cross
@@ -105,11 +105,7 @@ INSTALL_OVERRIDES=build_ocamldoc=false WITH_DEBUGGER= OCAMLRUN=ocamlrun
 .PHONY: installcross
 installcross:
        # Create dummy files to keep `install` happy
-       touch \
-         $(addprefix toplevel/, \
-           $(foreach ext,cmi cmt cmti cmx, native/nat__dummy__.$(ext)) \
-             all__dummy__.cmx topstart.o native/tophooks.cmi)
+       touch toplevel/topstart.o
        $(LN) `command -v ocamllex` lex/ocamllex.opt$(EXE)
-       $(LN) `command -v ocamlyacc` yacc/ocamlyacc.opt$(EXE)
        # Real installation
        $(MAKE) install $(INSTALL_OVERRIDES)

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Oct 22, 2025

I read through this PR, but not testing it (yet). I like very much the cleanup in the installation rules! I’ve taken a few notes, here and inlined.

  • In a0933f2, installation of native DLLs is removed: was that on purpose?

Oo, it's intentional (they should never have been being installed in the first place), but I'm not sure I should be doing it so subtly, indeed! It was a misunderstanding that crept in with the tsan changes, I think - we never need stub DLLs for native code (because .cmxs is already a stub DLL...)

  • In 79c7fea commit message, “relative” means “explicit”, I guess?

Yes - I should be explicit about explicitness, indeed, given that all the paths are relative!!

  • 515e861 creates a very elegant solution, congratulations :-)
  • 7fee32c: what a relief! :-)

Thank you 🙂

@shym
Copy link
Copy Markdown
Contributor

shym commented Oct 23, 2025

  • In a0933f2, installation of native DLLs is removed: was that on purpose?

Oo, it's intentional (they should never have been being installed in the first place), but I'm not sure I should be doing it so subtly, indeed! It was a misunderstanding that crept in with the tsan changes, I think - we never need stub DLLs for native code (because .cmxs is already a stub DLL...)

I thought it strange for them to be dropped without a mention in the commit message, even if I didn’t know when (that is to say, even just if) they were useful.

Maybe we could drop the byt in dll*byt* filenames (if that doesn’t break backward compatibility, though)?

@dra27 dra27 force-pushed the opam-install-file branch from 3aeff31 to 4eb9eec Compare November 9, 2025 08:27
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Nov 9, 2025

Rebased - review responses to follow

Copy link
Copy Markdown
Contributor

@shym shym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the range-diff and the “Review” commits (thank you so much yet again for such a clean update!): I managed to find only a missing comma :-D So this looks good to me!

Comment on lines +55 to +56
(* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec
and opam-man to produce [file] *)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(* [generate_install file] processes then erases opam-bin, opam-lib opam-libexec
and opam-man to produce [file] *)
(* [generate_install file] processes then erases opam-bin, opam-lib,
opam-libexec and opam-man to produce [file] *)

Copy link
Copy Markdown
Member

@damiendoligez damiendoligez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after interactive review; still some final testing to do.

@dra27 dra27 force-pushed the opam-install-file branch 2 times, most recently from 937f9f6 to 3c2f8c8 Compare November 30, 2025 18:07
@dra27 dra27 force-pushed the opam-install-file branch 2 times, most recently from c47392a to dc9a4fd Compare December 11, 2025 18:35
All of the .mli files in toplevel/byte are "common", so they're already
installed by the patterns in toplevel.
It's necessary to install main.o and optmain.o because the two modules
are not part of ocamlbytecomp.cmxa and ocamloptcomp.cmxa, but the .cmx
files are already installed as part of wildcard patterns on driver/
The ocamldoc installation commands in installopt duplicate a series of
commands which already happen in install.
The ocamldoc binaries were installed using a relative path. They're the
only binaries installed that way - switch them to use an implicit path,
as it's easier to make an implicit path subsequently explicitly relative
if needed than vice versa.
make install works as it did before. All the commands in the install
targets now go through a macro call which allows the semantic intent of
each command to be more clearly specified.
make [INSTALL_MODE=install] install
  - installs the compiler, as normal
make INSTALL_MODE=display install
  - displays the operations needed for
make INSTALL_MODE=list install
  - lists the files and symbolic links which are installed
@dra27 dra27 force-pushed the opam-install-file branch from 976bb76 to 76ef1bc Compare December 12, 2025 12:14
#* *
#**************************************************************************

set -e
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some extra safety maybe?

Suggested change
set -e
set -eu
if set -o pipefail > /dev/null 2> /dev/null; then
# POSIX.1-2024
set -o pipefail
fi

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question for the generated -clone and -fixup scripts

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, indeed - that's a good idea!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What am I missing in calling it twice? Isn't

set -o pipefail > /dev/null 2> /dev/null || true

sufficient?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yes, of course

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah - it turns out neither of us appears to have been correct! Apparently shell builtins are allowed to abort execution unconditionally, so it appears to be necessary to have:

if (set -o pipefail 2>/dev/null); then
  set -o pipefail
fi

With Ubuntu's default dash, both this:

#!/bin/sh

set -o pipefail || echo No pipefail

and

#!/bin/sh

if ! set -o pipefail; then
  echo No pipefail
fi

display an error from the set command but then immediately abort the script.

Copy link
Copy Markdown
Member

@kit-ty-kate kit-ty-kate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only took a couple of hours to briefly look at tools/opam/* but the general idea looks good, albeit a bit complex to decipher at first between all the various scripts and the planned opam-repository changes. I only had a couple of minor remarks that can be ignored if i was off (the opam --safe/readonly bits should probably be added though).

Hopefully opam will be able to clone relocatable packages itself in the future, which should allow to shave off a bit of the complexity by removing the clone mode, but that will probably need some time to be designed and implemented correctly.

Also the addition of the .install file is a really nice improvement i think.

#* *
#**************************************************************************

set -e
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question for the generated -clone and -fixup scripts

let _ =
let create_dir seen (dir, _, _) =
if not (StringSet.mem dir seen) && String.contains dir '/' then
output_endline oc {|mkdir -p '%s'|} dir;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably best to avoid the silent invariant that no ' character exists in dir

Suggested change
output_endline oc {|mkdir -p '%s'|} dir;
output_endline oc {|mkdir -p %s|} (Filename.quote dir);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That just replaces it with a different invariant on Windows (Filename.quote doesn't do the same thing). I think it's more hassle than it's worth, given that this structure is only the root directories of opam switches (lib/ocaml, bin, etc.)

String.map (function '@' -> '/' | c -> c)
(String.sub file 6 (String.length file - 6))
in
output_endline oc {|mkdir -p "$1/%s"|} dir;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
output_endline oc {|mkdir -p "$1/%s"|} dir;
output_endline oc {|mkdir -p "$1"/%s|} (Filename.quote dir);

In_channel.fold_lines (fun _ line ->
match String.split_on_char ' ' line with
| [source; dest] ->
output_endline oc {|cp '%s' "$1/%s/%s"|} source dir dest
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
output_endline oc {|cp '%s' "$1/%s/%s"|} source dir dest
output_endline oc {|cp %s "$1"/%s/%s|}
(Filename.quote source) (Filename.quote dir) (Filename.quote dest)


let clone_files oc dir ic =
output_endline oc
{|dest="$1/%s" xargs sh "$1/clone-files" <<'EOF'|} dir;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{|dest="$1/%s" xargs sh "$1/clone-files" <<'EOF'|} dir;
{|dest="$1"/%s xargs sh "$1/clone-files" <<'EOF'|} (Filename.quote dir);

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 12, 2025

I like the --safe idea - but I'd ruled out Filename.quote - as the quoting stands, it should be exactly quoting actual untrusted input the "$1", etc. while leaving names which we are entirely in control of on the build/install side not at risk of being misquoted by an unexpected platform interaction!

@dra27 dra27 force-pushed the opam-install-file branch from 76ef1bc to f416c48 Compare December 12, 2025 22:25
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 12, 2025

Thanks, @kit-ty-kate! --safe added to those calls; in re-checking the equivalence of the interim shell scripts and generate.ml, I spotted that the $CP fallbacks weren't being correctly escaped on Windows and there were a couple of unnecessary differences between the two versions.

I haven't changed the set mode on process.sh yet, just because I want to plug that into a test properly, and the "apparatus" for doing that is out-of-sync at the moment!

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 12, 2025

Separately, this has been tortured on precheck#1091 with a special check that builds the compiler and the trunk commit it sits on and compares the installation artefacts (i.e. checks that the same files are installed) and then repeats the entire process for --disable-suffixed. It seems to be passing!

@dra27 dra27 force-pushed the opam-install-file branch 2 times, most recently from 91d84b9 to b7b3efd Compare December 15, 2025 12:53
make INSTALL_MODE=opam install generates $OPAM_PACKAGE_NAME.install and
$OPAM_PACKAGE_NAME-fixup.sh ($OPAM_PACKAGE_NAME defaults to
ocaml-compiler). Nothing is installed by this mode. The fixup.sh script
is intentionally not made executable (it should be invoked explicitly
with sh) and creates symbolic links, if required, and also manually
copies the files to the doc dir, as the .install file format doesn't
allow the correct location to be specified.
Redundant filter, but mitigates an issue with --assume-built in opam
Mitigates an issue with opam install --assume-built since build
dependencies are ignored. The semantics should remain consistent: in
particular, as there is only a single version of each of these packages,
the key issue is that removing the package will still trigger the
correct behaviour as the dependency graph will change.
@dra27 dra27 force-pushed the opam-install-file branch from b7b3efd to f13b75e Compare December 15, 2025 14:02
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 15, 2025

Updated torture version going through precheck#1092. I've added a commit at the end of the series which adds guards in the Makefile-machinery to detect filenames which would cause problem, I've also - for the sake of history, and because it was a useful extra check - ensured that the shell and OCaml generators are actually producing the same scripts and hardened the use of single-quoting on all-generated filenames.

Makefile.common Outdated
# $(1) can be a mix of filenames and globs. Check $(1) itself for illegal
# characters (i.e. ensure that the whole of $(1) is a "valid" path) and then for
# each word containing a pattern character also evaluate the pattern and check
# the resulting files. $(2)/$(3) be be a valid section.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be bemust be maybe?

@dra27 dra27 force-pushed the opam-install-file branch from f13b75e to 114231d Compare December 17, 2025 13:55
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 17, 2025

Last couple of tweaking notes to those macros added. However, @shym and I discussed this separately - the benefit of the make macros is that it would catch errors just doing a normal make install, but that's not a normal part of a development workflow - given that CI is always going to be checking the opam/clone mechanisms, it therefore seems sufficient to keep the sanity checks in tools/opam/generate.ml and remove at least some of the meta-programming in Makefile.common!

Additional hardening on the requirements for names of files and
directories in the distribution.
@dra27 dra27 force-pushed the opam-install-file branch from 114231d to 7cf5ef2 Compare December 17, 2025 14:01
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 17, 2025

Assuming everything passes CI again here, and everyone remains happy, I think this is good to go, too. Relocatable and now easy to clone, too...!

@dra27 dra27 added the merge-me label Dec 17, 2025
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Dec 17, 2025

This happened on #14244 as well - the AppVeyor issue is that the "CI: Full matrix" check is occasionally pushing this over its 1 hour cancellation mark on the occasions when we get an over-stretched worker.

@NickBarnes NickBarnes merged commit f6c8af4 into ocaml:trunk Dec 17, 2025
37 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI: Full matrix Makes the CI test a bigger set of configurations merge-me no-change-entry-needed relocatable towards a relocatable compiler run-crosscompiler-tests Makes the CI run the Cross compilers test workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants