Skip to content

lib-pkg alternative to lib-ext#3190

Merged
AltGr merged 5 commits intoocaml:masterfrom
dra27:lib-pkg
Jan 26, 2018
Merged

lib-pkg alternative to lib-ext#3190
AltGr merged 5 commits intoocaml:masterfrom
dra27:lib-pkg

Conversation

@dra27
Copy link
Copy Markdown
Member

@dra27 dra27 commented Jan 25, 2018

This is a quite old piece of my porting from when updating dependent packages frequently was quite important for Windows changes, and fighting OCamlMakefile was tedious. The main thing this PR adds is the ability to do a normal installation of opam's dependencies directly to the compiler built by make compiler in bootstrap/. The advent of Jbuilder, which has been a more reliable replacement build system for lib-ext than OCamlMakefile was, has rendered this less useful than it was, but it's still quite handy.

This PR also introduces an extra Travis test which essentially tests make cold (and uses lib-pkg instead of lib-ext).

The bootstrap-ocaml.sh script is tweaked to write src_ext/Makefile.config which contains the necessary information for lib-pkg to compile. src_ext/Makefile is split into src_ext/Makefile.sources which contains the URLs and MD5s for all the tarballs and src_ext/Makefile.packages which contains the instructions needed to build each package from its sources tarball, and also the dependency graph between them. src_ext/Makefile is extended to permit alternating between make lib-ext and make lib-pkg mode. The src_ext/patches directory is also restructured to allow for patches needed in both modes and patches needed only in one mode.

There are a few other minor fixes included in this PR:

  • The patch to lib-ext CUDF, which was always weird in the first place, is no longer necessary, and in fact was causing the mccs testsuite to fail (because it results in the output being displayed in a different order!)
  • lib-ext is now set-up to disable any Jbuilder runtest aliases in the vendor'd packages - the mccs and cppo tests were also being run.
  • make lib-pkg assumes the package layout for the unreleased ocaml-mccs 1.1+5 (just because this has come from Windows development). It would probably be better to cut a release of ocaml-mccs before this is merged.

@AltGr
Copy link
Copy Markdown
Member

AltGr commented Jan 25, 2018

Thanks! I am not sure of the precise use-cases for it, but if it can help bootstrapping and building on Windows, it's great.
If I get it correctly, this will re-build lib-ext dependencies using their own build-systems (with patches) and install them separately ? If so, maybe it should be made clearer that this is an auxiliary setup tool, because building all the deps (possibly partially) with jbuilder directly from source, and ignoring their build systems, does feel much cleaner for the "normal" work-flow :)

There is of course still the third way, closer to lib-pkg actually, that installs all the dependencies using opam :p

dra27 added 2 commits January 26, 2018 09:20
If a bootstrap compiler is built, make lib-pkg may be run before
configure to build all the required dependencies as findlib packages as
an alternative to running make lib-ext after configure.

The retrieval of the archives has much in common with the lib-ext
method, and the two mechanisms are designed to live in harmony in the
same Makefile. The patches directory is now ordered so that
src_ext/patches/foo are the patches for foo when built with lib-ext
(so as before) and src_ext/patches/foo.pkg are the patches for foo when
built with lib-pkg. Additionally, src_ext/patches/foo.common are used
with both mechanisms.

The URLs and MD5 checksums have been moved to src_ext/Makefile.sources
and the build instructions for each package are placed in
src_ext/Makefile.packages.

src_ext/Makefile has been updated to work without ../Makefile.config
and Makefile cookies are used to ensure that the clone and clone-pkg
targets correctly set-up the build directories.

make lib-pkg supports parallel make.
Ubuntu 14.04 still ships with pre-4.0 make, which doesn't work with the
lib-pkg mode.
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Jan 26, 2018

That's correct - the only patches included are ones which are either upstreamed and not yet released or in the process of being upstreamed. It does indeed aim to be closer to the mode where you've used opam to install the dependencies - the difference is that lib-pkg is an extension of make cold/make compiler so it's (mildly) useful when you're developing on a system where you're developing using a bootstrapped compiler. There are also occasions (getting rarer as dune bugs are eliminated and new features added) when you want to blow away _build while developing... and at that point lib-ext is annoying, because you have to rebuild all the deps again! I'm also bad at keeping multiple opam roots organised (and I keep running out of space on my VMs), so I prefer not to have a system-wide opam around when I'm developing opam!!

The README for lib-pkg is in the temporary Windows instructions immediately after the bootstrapping instructions for getting a compiler, which I think is clear enough? I completely agree that now that we have a composable build system like Dune, there may be a magic point in the future where the lib-ext preparation is just untarring the release tarballs of the deps!

@AltGr
Copy link
Copy Markdown
Member

AltGr commented Jan 26, 2018

Sure, just wanted to be sure I got it correctly. Thanks!

@AltGr
Copy link
Copy Markdown
Member

AltGr commented Jan 26, 2018

PRed the mccs release to make way for this

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Jan 26, 2018

Hang on - I'll rebase with it!

dra27 added 3 commits January 26, 2018 10:09
Especially as it was causing the ocaml-mccs tests to behave differently
(unstable sorting).
Running cppo and mccs's testsuites otherwise!
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Jan 26, 2018

Right, let's see how long CI takes! Incidentally, @AltGr, I've been going through travis-ci.org on some of mine and looking at old pull request jobs and branch jobs and killing the long (not) running macOS jobs... it's slowly helping bring down the Travis backlog!

@AltGr AltGr merged commit afeca5c into ocaml:master Jan 26, 2018
@dra27 dra27 deleted the lib-pkg branch April 4, 2018 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants