Embed cygwin's setup.exe on windows as a fallback when cygwin.com is unavailable by kit-ty-kate · Pull Request #6526 · ocaml/opam

kit-ty-kate · 2025-06-02T01:56:41Z

Supersedes #6523
Fixes #6498
related #6474

dra27 · 2025-06-02T12:28:21Z

configure.ac

+AC_ARG_WITH([embed-cygwin-setup],
+  AS_HELP_STRING([--embed-cygwin-setup],["Embed Cygwin's setup.exe into the executable"]))
+
+AS_IF([test "x${with_embed_cygwin_setup}" = "xyes"],[
+  curl --silent -Lo cygwin-setup.exe https://cygwin.com/setup-x86_64.exe
+  ocaml shell/embed.ml cygwin-setup.exe > src/state/opamEmbedCygwinSetup.ml
+])
+


embedded rather than embed? So --with-embedded-cygwin-setup (or it could just be --with-cygwin-setup?):

Suggested change

AC_ARG_WITH([embed-cygwin-setup],

AS_HELP_STRING([--embed-cygwin-setup],["Embed Cygwin's setup.exe into the executable"]))

AS_IF([test "x${with_embed_cygwin_setup}" = "xyes"],[

curl --silent -Lo cygwin-setup.exe https://cygwin.com/setup-x86_64.exe

ocaml shell/embed.ml cygwin-setup.exe > src/state/opamEmbedCygwinSetup.ml

])

AC_ARG_WITH([embedded-cygwin-setup],

AS_HELP_STRING([--with-embedded-cygwin-setup],["Embed Cygwin's setup.exe into the executable"]))

AS_IF([test "x${with_embedded_cygwin_setup}" = "xyes"],[

curl --silent -Lo cygwin-setup.exe https://cygwin.com/setup-x86_64.exe

ocaml shell/embed.ml cygwin-setup.exe > src/state/opamEmbedCygwinSetup.ml

])

In passing, to avoid compulsory network access in configure when using this, it'd be tempting to support --with-embedded-cygwin-setup meaning to download and --with-embedded-cygwin-setup=setup-x86_64.exe to use a local copy

rjbou · 2025-06-02T17:05:10Z

shell/embed.ml

@@ -0,0 +1,22 @@
+let add_chan buf chan =


(* General module to embed content. Usage: 'ocaml shell/embed.ml > opamEmbedModule.ml' *) let add_chan buf chan =

rjbou · 2025-06-02T17:06:44Z

configure.ac

 AC_ARG_WITH([vendored-deps],
  AS_HELP_STRING([--with-vendored-deps],[Use vendored dependencies instead of any available in the compiler installation]))

+AC_ARG_WITH([embed-cygwin-setup],


Do we want to make this option available only on Windows system ?

rjbou · 2025-06-02T17:10:49Z

configure.ac

+  AS_HELP_STRING([--embed-cygwin-setup],["Embed Cygwin's setup.exe into the executable"]))
+
+AS_IF([test "x${with_embed_cygwin_setup}" = "xyes"],[
+  curl --silent -Lo cygwin-setup.exe https://cygwin.com/setup-x86_64.exe


Do we want to check the hash ?

rjbou · 2025-06-02T17:13:56Z

shell/embed.ml

+    {|(* THIS FILE IS AUTOMATICALLY GENERATED, EDIT configure.ac & shell/embed.ml INSTEAD *)
+let content = Some "%s"
+|}
+    (String.escaped content_setup)


We loose some compression by not using base64 (2.1M vs 1.5M)

That's not what i'm observing. Base64 doesn't compress; on the contrary as far as i know it makes the data bigger due to its encoding.
Test done encoding my local /usr/bin/cat ELF binary:

base64: 1.87MB

String.escaped: 1.77MB

Erm, apples and oranges are being compared here - the base64 version was writing an actual base64 string into the executable - so the setup executable will be 33% larger. This version is writing the OCaml string constant to the source file ... so the data in the executable will be a verbatim copy of the original executable (plus a tiny bit of overhead from the marshal header)

Indeed, I tested the base64 tool and diffed the generated file, not the library directly in the code.

src/state/opamEmbedCygwinSetup.mli

rjbou · 2025-06-03T16:46:51Z

Good to go once the local test validated

dra27

Tested - both for opam init and opam update --depexts and also the configure option with a local copy of an older setup.exe ... it's working beautifully!

A couple of minor suggestions in configure.ac, a couple of totally optional thoughts on the crunching... the log/warning at init is the only thing which I think ought to be done (in particular, it means that CI runs which happen to use a cached copy should clearly display it, etc.)

dra27 · 2025-06-03T20:18:09Z

configure.ac

+AS_IF([test "x$with_cygwin_setup" != "x"],[
+  AS_CASE([$with_cygwin_setup],
+    [yes], [
+      curl --silent -LO https://cygwin.com/sha512.sum


The file includes checksums for the 32-bit setup program which causes this:

shasum: setup-x86.exe: No such file or directory setup-x86.exe: FAILED open or read setup-x86_64.exe: OK shasum: WARNING: 1 listed file could not be read

Possibly just:

Suggested change

curl --silent -LO https://cygwin.com/sha512.sum

curl --silent -L https://cygwin.com/sha512.sum | grep -F x86_64 > sha512.sum

which then has:

checking OCaml Sys.os_type... Win32 setup-x86_64.exe: OK checking for gawk... gawk

dra27 · 2025-06-03T20:19:10Z

configure.ac

+      CYGWIN_SETUP=setup-x86_64.exe
+    ],
+    [no], [],
+    [CYGWIN_SETUP=$with_cygwin_setup])


Means that a Cygwin path can be used to specify the file (e.g. $PWD/cache, etc.)

Suggested change

[CYGWIN_SETUP=$with_cygwin_setup])

[CYGWIN_SETUP="$(cygpath -m "$with_cygwin_setup")"])

dra27 · 2025-06-03T20:25:45Z

src/state/dune

  (deps    ../../shell/crunch.ml (glob_files shellscripts/*.*sh))
  (action  (with-stdout-to %{targets} (run ocaml %{deps}))))
+
+; Embed Cygwin mechanism (done in the configure script on Windows if requested)


Two completely optional observations, just having spotted it: Embedded, rather than Embed? It could be processed in crunch.ml, but it's not like adding a module particularly matters?

crunch.ml does something slightly different with regards to \n so i'd rather not overcomplicate this PR. We can definitely look into merging the two later though if you want. If you're fine with that i'll open a separate issue to fix later.

dra27 · 2025-06-03T20:27:09Z

src/state/opamSysInteract.ml

+         else
+           begin match OpamEmbedCygwinSetup.content with
+           | None -> Printexc.raise_with_backtrace exn backtrace
+           | Some setup_exe_content -> OpamFilename.write dst setup_exe_content


I think it ought to at least display a log entry that it wrote from an internal copy, or even better a warning, e.g.

Suggested change

| Some setup_exe_content -> OpamFilename.write dst setup_exe_content

| Some setup_exe_content ->

OpamConsole.warning "Cygwin setup Failed to download - using embedded copy instead";

OpamFilename.write dst setup_exe_content

kit-ty-kate · 2025-06-03T20:58:04Z

@dra27 i believe that's everything included

dra27 · 2025-06-04T08:51:17Z

There was a typo in the change that meant setup was no longer downloaded - I spotted the "unclean" files while checking that and, as I'm on a train, also happened to spot that it ignores errors if the download fails, which is the backdrop for the other two commits!

…unavailable Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com> Co-authored-by: David Allsopp <david.allsopp@metastack.com>

dra27 · 2025-06-05T06:31:16Z

Spectrum is green!

kit-ty-kate added this to the 2.4.0~alpha3 milestone Jun 2, 2025

kit-ty-kate added the AREA: PORTABILITY label Jun 2, 2025

rjbou self-requested a review June 2, 2025 12:10

dra27 reviewed Jun 2, 2025

View reviewed changes

rjbou reviewed Jun 2, 2025

View reviewed changes

kit-ty-kate force-pushed the embed-cygwin branch 2 times, most recently from 82a6d99 to 88711e8 Compare June 3, 2025 13:07

kit-ty-kate requested a review from rjbou June 3, 2025 14:43

kit-ty-kate marked this pull request as ready for review June 3, 2025 14:43

rjbou reviewed Jun 3, 2025

View reviewed changes

src/state/opamEmbedCygwinSetup.mli Show resolved Hide resolved

kit-ty-kate force-pushed the embed-cygwin branch from 88711e8 to 4162d89 Compare June 3, 2025 16:34

rjbou approved these changes Jun 3, 2025

View reviewed changes

dra27 approved these changes Jun 3, 2025

View reviewed changes

kit-ty-kate force-pushed the embed-cygwin branch from 4162d89 to 3c28f71 Compare June 3, 2025 20:52

kit-ty-kate force-pushed the embed-cygwin branch from 7b2b919 to 4ad129b Compare June 4, 2025 18:48

kit-ty-kate and others added 2 commits June 5, 2025 06:10

Embed cygwin's setup.exe on Windows as a fallback when cygwin.com is …

e7fe3dc

…unavailable Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com> Co-authored-by: David Allsopp <david.allsopp@metastack.com>

Fix Alpine 22 depexts check

a9131b0

dra27 force-pushed the embed-cygwin branch from 7efeb39 to a9131b0 Compare June 5, 2025 05:11

dra27 merged commit fee851c into ocaml:master Jun 5, 2025
56 checks passed

kit-ty-kate deleted the embed-cygwin branch June 5, 2025 11:59

This was referenced Jun 5, 2025

cygwin: embed setup.exe to use it as a fallback as a last resort #6523

Closed

Release: Call the opam configure script with on Windows #6538

Merged

kit-ty-kate mentioned this pull request Jun 19, 2025

./configure --with-cygwin-setup requires perl #6557

Closed

jmgrosen mentioned this pull request Jan 16, 2026

feat(windows): use opam's internal cygwin installation semgrep/setup-ocaml#17

Merged

	curl --silent -LO https://cygwin.com/sha512.sum
	curl --silent -L https://cygwin.com/sha512.sum \| grep -F x86_64 > sha512.sum

	[CYGWIN_SETUP=$with_cygwin_setup])
	[CYGWIN_SETUP="$(cygpath -m "$with_cygwin_setup")"])

Conversation

kit-ty-kate commented Jun 2, 2025 • edited by rjbou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dra27 Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rjbou commented Jun 3, 2025

Uh oh!

dra27 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kit-ty-kate commented Jun 3, 2025

Uh oh!

dra27 commented Jun 4, 2025

Uh oh!

dra27 commented Jun 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kit-ty-kate commented Jun 2, 2025 •

edited by rjbou

Loading

dra27 Jun 2, 2025 •

edited

Loading