Use dune to build unikernels by TheLortex · Pull Request #1226 · mirage/mirage

TheLortex · 2021-05-05T12:45:59Z

This is hopefully the final attempt for the integration of dune in Mirage. There are various documents scattered around this repository explaining the changes. For reference:

PR#1: Use dune to build unikernels #979
PR#2:dunification of MirageOS #1020
PR#3: Dunification of MirageOS #1024
PR#4: Use dune to build the unikernel #1153
Tracking issue: mirage 4.0 tracking issue #1217

You should be able to test this PR using the following documentation: https://next.mirage.io/wiki/mirage-4
It uses https://github.com/mirage/mirage-dev/tree/mirage-4 for packages that haven't been upstreamed yet.

Goal

The goal is to locally fetch unikernel dependencies and use dune to build unikernels, taking advantage of the dune-workspace (https://dune.readthedocs.io/en/stable/dune-files.html#dune-workspace) file to define and customize mirage targets.

Only build tools are to be installed in the opam switch, the unikernel runtime dependencies are locally fetched using the opam-monorepo tool: https://github.com/ocamllabs/opam-monorepo. opam-monorepo lacks some features needed for mirage but they will be upstreamed and a release will be made before Mirage 4 comes out: in the meantime the required patches have been aggregated in my personal repository: https://github.com/TheLortex/duniverse/tree/mirage-4.

PR changes

I've split the changes into small commits for increased readability, but the commits are not self-contained (i.e. build might fail between two commits).

Functoria: add Action.run_cmd_cli: When config.exe or dune is run it's nice to keep the interactive terminal capabilities. So the Action.run_cmd_cli action behaves similarly to Action.run_cmd but without touching stdin or stdout.
Functoria: separate installation kind for packages: Build tools need to be installed in the switch (ocaml-freestanding, solo5, ..) and runtime libraries to be locally fetched (mirage-solo5, mirage-runtime, ...), so we separate the two kind of packages using a Global | Local variant. Two opam files are generated, one consumed by opam and the other consumed by opam-monorepo.
Functoria CLI: add extra-repo option: Not all packages build with dune. For the others, an overlay opam repository has to be used in order to make them compatible with mirage. The --extra-repo option takes care of adding a custom overlay repository when fetching the dependencies. By default it's https://github.com/mirage/opam-overlays
Functoria: file generation only happen configure-time: At build time, a functoria device can use dune rules to generate files.
Functoria: move context cache in <build_dir>/ <P.name> /context: Instead of mirage.context: the goal is to reduce the amount of files generated in the unikernel folder. For that matter, most files will be generated in <build_dir>/mirage
Functoria: makefile -> use opam-monorepo to locally fetch deps: Update Makefile generation
Functoria: add Dune helper module to generate dune files / devices can have dune rules to generate files at build time / generate dune files, use dune to build unikernels: Generating dune files to build applications.
Functoria: update app_info device: adapt to previous changes
Mirage: adapt to functoria changes
Mirage: target interface changes: the interface changes a bit, now that targets have to issue dune rules and workspace configuration to build unikernels
Mirage: adapt targets to dune: target-specific rules to build unikernels using dune
Mirage: port devices to dune: devices such as crunch now issues dune rules instead of build-time actions

Now that the dune build layout is approved and tested, it believe it's important to have it merged so that people other than me can get used to it and be able to contribute directly. If you have any remarks on any part of the changes I'll glad to hear them and make appropriate updates.

For CI: mirage/mirage-dev#345 / mirage/mirage-skeleton#307

lib/functoria/cli.ml

Drup · 2021-05-07T11:15:06Z

Is that the final final one? :)

I'm not completely sure about having dune stanza in functoria devices, but maybe tying functoria with dune is the best way forward. It certainly seems like a working solution! For the rest, it looks good to me.

dinosaure · 2021-05-10T08:58:18Z

Can you rebase your PR with master? It includes the patch to fix OCaml-CI (with alcotest.1.4.0).

TheLortex · 2021-05-10T09:14:26Z

Hello @Drup, yes indeed this PR specializes functoria for the mirage use-case by making it dune-specific. However I think that adding a new layer of abstraction (the build system) would make the changes a bit more complicated, but that's something we can still explore in the future. We could make functoria (and thus mirage) more build/package-system independent: we could imagine other "build backends" such as esy or nix ! But for now, let's stick to the opam-monorepo / dune tooling for the sake of simplicity

Drup · 2021-05-10T09:56:05Z

I'm not opposed to it, on the contrary. My very initial worry was in the context where there was no "dominant" tool, and the most used one wasn't terribly satisfactory. This is not really the case anymore. :)

dinosaure · 2021-05-10T14:54:12Z

As far as I can see, the PR is good for me. The CI is correct and we tested it in few unikernels this branch (so, yes @Drup, this is the last one 😃). We should have a double-check for someone else (/cc @mirage/core). Then, we can really start the process release of MirageOS 4 🎉.

yomimono · 2021-07-16T16:20:51Z

If there are specific tasks or tests required to help with merging and releasing this, please let me know. I find myself with (1) a bit of extra time and (2) unikernels I'd like to build within a larger Dune project, and (2) doesn't work well with the current released tooling.

CHANGES.md

samoht · 2021-07-16T16:42:04Z

lib/functoria/filegen.ml

    | `Sexp -> Fmt.str ";; Generated by"
    | `Opam | `Make -> "# Generated by"
    | `OCaml -> "(* Generated by"
+    | `OpamLock -> "synopsis: \"duniverse generated"


why does headers and short_headers disagree on what to do for OpamLock?

Also: OpamLock doesn't seem to be used elsewhere?

headers is used when Mirage generates files.
short_headers is used to identify files generated by Mirage.

For the opam lockfile, only the identification function (short_headers) was used to remove the file in the clean step. Now that most mirage files (include the lockfile) are stored together inside the mirage/ folder, there's no need to identify OpamLock anymore so it can be removed

lib/functoria/package.mli

samoht · 2021-07-16T16:51:55Z

test/mirage/query/run.t

  ]


+Query local opam


So by reading that example, I understood the local/global distinction wrongly. @TheLortex can you try to clarify this a bit?

What's the difference between the local/global/install/locked opam files? :-)

There are three opam files in a mirage project:

<name>.opam: unikernel depencies that have to be locally fetched so that they exist in the dune workspace. Notably all runtime dependencies should be in that opam file, but it can also contain build tools as long as they build using dune.

<name>.opam.locked: unikernel dependencies lockfile obtained using opam-monorepo.

<name>-install.opam: the rest of the build tools, the main example being ocaml-freestanding. They are installed in the switch using opam.

I agree that the names are a bit confusing but I don't know any other option. The rationale is that there are dependencies that need to be installed globally in your opam switch and that there are runtime dependencies that need to be locally fetched alongside the unikernel files as a monorepo in a dune workspace.

New proposal:

global becomes switch

local becomes monorepo
Therefore the names are more in sync with the traditional ocaml concepts (opam switch and opam-monorepo)

What do you think ?

dinosaure · 2021-07-16T17:11:47Z

If there are specific tasks or tests required to help with merging and releasing this, please let me know. I find myself with (1) a bit of extra time and (2) unikernels I'd like to build within a larger Dune project, and (2) doesn't work well with the current released tooling.

Currently, we have an unofficial explanation from @TheLortex about MirageOS 4 which is available here: https://next.mirage.io/wiki/mirage-4. It will be nice if you can reproduce the process locally and give to us some feedbacks 👍 - it will help us a lot to catch missing steps/states/etc.

PS: note that the website is deployed with MirageOS 4 ☺️

yomimono · 2021-09-10T19:02:40Z

With a couple small changes I was able to build mirage-www on OCaml 4.12 with the mirage-4 development repository. I'm able to run the unikernel via hvt as in the simple instructions. 🎉

TheLortex · 2021-09-27T14:35:03Z

Some updates on the PR:

Improve the changelog description
I've renamed the opam file distinction local/global to monorepo/switch to improve clarity
Package.kind is now Package.scope
Now that opam 2.1 is out, I've removed the 2.0 support to keep the generated Makefile simple.
Removed the run ['_build/default/./config.exe' ...]: exited with 1 message when a command would fail

Mirage CI is still using 2.0 so I'll update that

samoht · 2021-10-07T18:20:03Z

Can you rebase/cleanup against master? I think it's good to merge.

I have extra changes to run e2e tests as cram tests, I can add those after the merge to avoid more conflicts.

samoht · 2021-10-08T13:56:04Z

I'm rebasing/cleaning up this PR.

the mirage-type deletion is in Remove mirage-types #1232

samoht · 2021-10-08T16:05:14Z

It's all green! @dinosaure or @TheLortex do you want to have a last review?

dinosaure · 2021-10-08T16:06:57Z

I would like to double-check ci.mirage.io and see if all works but seems good (even if you lost the history).

samoht · 2021-10-08T16:08:56Z

even if you lost the history

Yea it was done on purpose, as the various commits didn't compile separately and were intertwined :-)

It's also fine to make other PRs to fix the remaining issues (if needed)

TheLortex · 2021-10-08T16:10:37Z

I've juste updated mirage-dev's mirage-4 branch to the latest commit, let's see the result of the mirage 4 CI!

samoht · 2021-10-08T16:16:20Z

@TheLortex is the workflow to a PR documented somewhere? Do you think we should keep that workflow once 4.00 is released or it's just a temporary workflow? Maybe we want this to happen for every commit to main?

dinosaure · 2021-10-08T16:18:12Z

@TheLortex is the workflow to a PR documented somewhere? Do you think we should keep that workflow once 4.00 is released or it's just a temporary workflow? Maybe we want this to happen for every commit to main?

I think we should delete the MirageOS 3.0 support then and keep only one pipeline about MirageOS 4.

TheLortex · 2021-10-08T16:19:38Z

The pipeline is described here: https://github.com/ocurrent/mirage-ci
I'm working on fixing it. I'll keep a MirageOS 3 pipeline to watch the 3 branch

TheLortex · 2021-10-08T16:31:24Z

So to be clear, the CI tests this PR along with mirage/mirage-dev#345 and mirage/mirage-skeleton#303

Runs a command with stdout and stdin redirected to it. Allows keeping interactive terminal capabilities.

dinosaure · 2021-10-08T17:39:11Z

I rebased the PR to include #1233, let's see ci.mirage.io, I think we need an update from you @TheLortex?

TheLortex · 2021-10-08T17:40:47Z

No the CI stuff has been fixed, now it's all automatic :D

samoht · 2021-10-08T18:34:45Z

pgx is now failing with:

File "duniverse/ocaml-conduit/src/conduit-mirage/dune", line 10, characters 65-77:
10 |    ipaddr-sexp vchan tls tls-mirage xenstore.client uri.services ca-certs-nss))
                                                                      ^^^^^^^^^^^^
Error: Library "ca-certs-nss" not found.

(I think @hannesm fixed it recently?)

And block is failing with:

File "config.ml", line 11, characters 8-13:
11 |   impl ~build ~clean "Functoria_runtime" shellconfig
             ^^^^^
Error: The function applied to this argument has type
         ?packages:Mirage.package list ->
         ?packages_v:Mirage.package list Functoria__.Key.value ->
         ?install:(Functoria__.Info.t -> Functoria__.Install.t) ->
         ?install_v:(Functoria__.Info.t ->
                     Functoria__.Install.t Functoria__.Key.value) ->
         ?keys:Functoria__.Key.t list ->
         ?extra_deps:Mirage.abstract_impl list ->
         ?connect:(Mirage.info -> string -> string list -> string) ->
         ?dune:(Mirage.info -> Functoria__.Dune.stanza list) ->
         ?configure:(Mirage.info -> unit Functoria__.Action.t) ->
         ?files:(Mirage.info -> Fpath.t list) -> 'a Functoria__Impl.t
This argument cannot be applied with label ~build

dinosaure · 2021-10-08T18:35:44Z

Let's try the ci.mirage.io with this PR: mirage/mirage-skeleton#307

dinosaure · 2021-10-08T18:52:55Z

pgx is now failing with:

File "duniverse/ocaml-conduit/src/conduit-mirage/dune", line 10, characters 65-77:
10 |    ipaddr-sexp vchan tls tls-mirage xenstore.client uri.services ca-certs-nss))
                                                                      ^^^^^^^^^^^^
Error: Library "ca-certs-nss" not found.

(I think @hannesm fixed it recently?)

And block is failing with:

File "config.ml", line 11, characters 8-13:
11 |   impl ~build ~clean "Functoria_runtime" shellconfig
             ^^^^^
Error: The function applied to this argument has type
         ?packages:Mirage.package list ->
         ?packages_v:Mirage.package list Functoria__.Key.value ->
         ?install:(Functoria__.Info.t -> Functoria__.Install.t) ->
         ?install_v:(Functoria__.Info.t ->
                     Functoria__.Install.t Functoria__.Key.value) ->
         ?keys:Functoria__.Key.t list ->
         ?extra_deps:Mirage.abstract_impl list ->
         ?connect:(Mirage.info -> string -> string list -> string) ->
         ?dune:(Mirage.info -> Functoria__.Dune.stanza list) ->
         ?configure:(Mirage.info -> unit Functoria__.Action.t) ->
         ?files:(Mirage.info -> Fpath.t list) -> 'a Functoria__Impl.t
This argument cannot be applied with label ~build

So it seems fixed via my PR on mirage-skeleton according to the CI, @TheLortex currently rebased correctly all the stuff, I think we are ok to merge (just waiting the CI again) this PR 🎉 .

dinosaure · 2021-10-08T19:07:51Z

@samoht you can merge when you want, the CI is green 🌲

samoht · 2021-10-08T21:05:08Z

Welcome to MirageOS 4.0 :p

smorimoto · 2021-10-08T22:06:22Z

Woohoo!

yomimono · 2021-10-08T23:22:28Z

HELL YEAH

samoht reviewed May 5, 2021

View reviewed changes

lib/functoria/cli.ml Outdated Show resolved Hide resolved

dinosaure mentioned this pull request May 5, 2021

mirage 4.0 tracking issue #1217

Closed

29 tasks

This was referenced May 6, 2021

Use dune to build the unikernel #1153

Closed

Adapt to Mirage + dune mirage/mirage-skeleton#303

Merged

TheLortex force-pushed the mirage-4 branch from 3add43c to 3327917 Compare May 10, 2021 09:01

TheLortex force-pushed the mirage-4 branch from 3327917 to c177562 Compare May 10, 2021 09:52

sternenseemann mentioned this pull request Jul 3, 2021

Package MirageOS NixOS/nixpkgs#23955

Open