Nice work!

The only problem today about dune is: we can not use (dirs ...) to load some specific underlying directories into the dune.build file. I don't know why. The current trick is that mirage configure rewrite not-only dune.build but dune too to load correctly files needed to compile unikernel.

Do you have an example where you need this? This is supposed to be working fine I guess.

I'll have a proper review tomorrow.

I like the overall approach. As a quick response, I think that with:

And from that, dune can provide a well-defined workspace with right C flags according the target to let the user to use C stubs without the hard plumbing needed when we want to package that to OPAM.

That this should never happen:

An unikernel for Solo5 which links with digestif.c (compiled to be used by an UNIX executable) can work.

If we just compile every target in a well defined workspace, then dune caching in dune 2.0 can make it as fast, without any dangerous sharing involved.

Do you have an example where you need this? This is supposed to be working fine I guess.

By facts, we need to put a (dirs ...) into dune instead dune.build when we want to have a Solo5 unikernel. (dirs ...) wants to integrate some generated files by mirage configure -t hvt available in .mirage.solo5 directory - so you can view (dirs .mirage.solo5). These flags are cflags.sexp, lflags, etc.

Into an higher-level, the question is already opened on mirage/ocaml-solo5#66 where:

• in this PR, mirage will do the call to pkg-config to get flags
• in the ocaml-freestanding PR, the ocaml-freestanding will provides these files

As @hannesm said, the ocaml-freestanding PR finally propose 2 ways (pkg-config and files) to get flags. May be it's wrong. In other side, mirage-os-xen provides files and the mirage tool use them - but it can, in the same way than a Solo5 target, generate by itself these files.

I do prefer having explicit files with the flags rather than invocations to pkg-config. The reason is that this makes tracking build systems rules far easier, instead of the environment variables we have to propagate for pkg-config. I'm happy to port the mirage-xen pieces to install files with flags (some already are installed, but possibly not completely).

I do prefer having explicit files with the flags rather than invocations to pkg-config. The reason is that this makes tracking build systems rules far easier, instead of the environment variables we have to propagate for pkg-config. I'm happy to port the mirage-xen pieces to install files with flags (some already are installed, but possibly not completely).

Ok good, from what I know, mirage-os-xen does not need an update about this point. If we want to follow this way about ocaml-freestanding, mirage/ocaml-solo5#66 need to be merged. By this way, (dirs ...) will not be a problem!

It seems that this last way can remove all occurrence of pkg-config into the mirage tool - however, we still need to figure out about libraries which has C stubs and want to be packaged into OPAM. This last point is out of the scope of these PR - and, currently, ocamlfind take the relay on that with linkopts field.

we still need to figure out about libraries which has C stubs and want to be packaged into OPAM

Yes, mostly this the xen support libraries, and in those case we adapt them all via files containing CFLAGS or some linkopts META file as you suggest.

I did a new branch clean-dunification-without-pkg-config available on my repository and diff is available here: #1021. Please I would like to continue the discussion here, I just would like to see the status of Travis and give you an easy way to see the patch.

From what I did, if we integrate mirage/ocaml-solo5#66, we mostly can delete any use of pkg-config into the mirage tool which can be good but if we do the choice, we will be stick on a less expressive mechanism to get right flags and static libraries.

At this stage and to avoid an overlap with #1018, do we want to use files or pkg-config to shape the dunification of MirageOS. We can imagine that the design with files works (but limitation can be reach easily, I think, in a near future). From what I know, nobody has a strong opinion on that, but we should keep in our mind that this solution can come with:

• a double maintenance cost (with pkg-config for libraries, and files for the mirage tool)
• a problem with we want to tune compilation of an unikernel

@samoht highlighted some problems about the linking:

• ocplib-endian comes with unix (we should use https://gihtub.com/dune-universe/ocplib-endian.git#duniverse-1.0` (some others packages have this problem like sexplib)
• cstruct.5.0.0 must be used (old version comes with unix)
• sexplib should not be a dependency where it was used only to pretty-print s-expression
• pkg-config should be removed when underlying packages like targets should not be available first - in other words, opam install mirage && mirage configure -t hvt should work (eg. dunification of MirageOS without pkg-config #1021 for implementation)

About dependencies on unix, I will add (forbidden_libraries unix) into dune.build generated for Solo5 and Xen targets to give you a proper error message (instead a link error).

forbidden_libraries was added into the generated dune.build file for Xen and Solo5 target here: af028f9 and it will be cherry-pick into this PR. The PR without pkg-config is better when the suppression of pkg-config is quasi-total, only Xen was not fixed about that.

The change is available in this patch: d18c02b.

Into details, flags provided by ocaml-freestanding is not very consistent where libmirage-solo5_bindings.a must be added into libs.sexp even if this static library is not a part of ocaml-freestanding (cc @mato). So I updated them to add libmirage-solo5_bindings.a, then dune.build with a specific rule copies the static library according the environment (OPAM environment or dune environment - the second is priority).

A mirage configure && mirage build or a mirage configure -t hvt && mirage build works with mirage-skeleton with OCaml 4.07.1.

Ok, good news, from clean-dunification-without-pkg-config, only 2 packages need an update and MODE=hvt make into mirage-skeleton works 🎉 !

• nocrypto depends on bigarray: I did a new branch into Move to bigarray-compat including patch from opam-repository dinosaure/ocaml-nocrypto#1 with the smallest patch. According current status of nocrypto, I advise to add an other patch to opam-repository.
• sexplib depends on bigarray: I did Use bigarray-compat instead bigarray to avoid a link with unix janestreet/sexplib#40
• ocplib-endian depends on bigarray (https://github.com/dune-universe/ocplib-endian.git#duniverse-1.0)
• gmp-freestanding must rename libgmp-freestanding.a to libgmp.a

@dinosaure: Thanks for working on this. It's a complex change with many moving pieces, and I'm not sure I understand all the changes you are making. I'm going to try and reply to the various points you've raised, but will also ask questions that may have already been answered elsewhere, or seem "obvious" to others, so please bear with me.

This is the purpose of the package gmp-freestanding and gmp-xen. My only concern about this way is to rename libgmp_freestanding.a and libgmp_xen.a to libgmp.a and let the compiler to choose these implementations according the target.

If I understand this correctly, you are proposing that:

For a package which requires specially ("cross") compiled C libraries and a target, where target is, for example, xen or freestanding (a.k.a. Solo5):

Instead of package providing a library lib<package>_<target>.a and mucking about with META files to force the linker to use the non-standard library name, package will provide lib<package>.a in some target-specific location and something (possibly ocaml-freestanding, but see below) will provide information to mirage needed to set the linker's library search path to use only(!) libraries from the target-specific location(s).

Correct?

Considering flags, Solo5 and Xen can provide in some ways -L$(opam config var lib)/gmp-{freestanding,xen} as a dependency. NOTE: that means for any compilation of any unikernels, -L shoud always be emitted by mirage-os-xen or ocaml-freestanding even if the unikernel does not depend on gmp

Without pkg-config we lose the ability to compose -L search paths -- what you write above seems to require ocaml-freestanding to know about every possible package in order to be able to emit the -L$(opam config var lib)/<package>. That does not scale, and is the biggest limitation of dropping pkg-config altogether.

As an alternative, could we instead have_package_-target install its libraries into a shared directory, such as $(opam config var lib)/mirage-<target>/lib, and this would be added to -L/provided to mirage by some suitable package? I write "some suitable" deliberately as this is probably not ocaml-freestanding's job, but should instead be done by mirage-<target>, i.e. mirage-xen or `mirage-solo5.

Related to this:

Into details, flags provided by ocaml-freestanding is not very consistent where libmirage-solo5_bindings.a must be added into libs.sexp even if this static library is not a part of ocaml-freestanding (cc @mato). So I updated them to add libmirage-solo5_bindings.a, then dune.build with a specific rule copies the static library according the environment (OPAM environment or dune environment - the second is priority).

That's because libmirage-solo5_bindings.a comes from a pkg-config .pc file: https://github.com/mirage/mirage-solo5/blob/master/lib/bindings/mirage-solo5.pc which is then used by mirage as the actual package queried from pkg-config: https://github.com/mirage/mirage/blob/master/lib/mirage_link.ml#L8. mirage itself currently does not ask for ocaml-freestanding explicitly, only as a pkg-config requirement of mirage-solo5.

Could the solution I write about further above, using a single, shared target-specific directory for C libraries be made to work for mirage-solo5 also? How does dune determine the name of the C library it will link in? For a (OCaml? OPAM?) package foo, is it always lib<foo>.a?

A new tool was added into mirage: cc-to-opt or cc-to-ocamlopt which will takes C flags and do a translation of -L to -I and -l to -cclib -l. It wants to resolve static libraries and say that they exists according all given path by -L. Then, it re-order options due to the precedence of -L over -l (-LA -la -LB and -LA -LB -la don't have the same behavior if liba.a is available into A and B - this behavior is pretty-close to the GMP case where libgmp.a is available into several directories).

Three questions:

1. Do we need this tool at all? Couldn't ocaml-freestanding or mirage-solo5 generate the information required in a format already suitable for passing to ocamlopt?
2. Relying on directory precedence only in the presence of multiple libraries with the same name is incredibly fragile and will come back to bite us in the future. Can't we have one and one only specific search path per xen/freestanding/...?
3. If you are using ocamlopt to link instead of ld, does that mean you are no longer passing -nostdlib? That would be bad, see above. The unikernel must not accidentally link in host libraries.

The previous point want to serve an other issue about the expected flags used to compile C sources (like GMP or digestif). An unikernel for Solo5 which links with digestif.c (compiled to be used by an UNIX executable) can work. That means that silently, we can link with the wrong libdigestif.a and the unikernel still works.

Again, this seems like almost but not quite the same issue as above? Sorry if I'm repeating myself, but it's important. We need a way to ensure that the toolchain is using -nostdlib and an explicit target-specific list of search path(s) only. Is this not possible?

Link step

As @samoht asked, I delayed the resolution of static libraries compiled with specific flags according the target to mirage build (instead mirage configure). A command-line like:

$ opam install mirage && mirage configure -t hvt

Should works. However, make depends still is needed before a mirage build. Computation still uses cc-to-opt small pass to correctly be more predictable about which static libraries will be linked. However, this pass still be on top of what META files give to us.

A new file appears under the hood extra-libraries.sexp which is empty at the beginning and then was filled by pkg-config --static --libs %{target} and ocamlfind query -predicate native -format '%d %(%{target}_linkopts) to still be happy with the current status of the MirageOS eco-system.

In others words, pkg-config and ocamlfind still are needed.

A new flag was added (-noautolink) into the link command only for Solo5 and Xen target when ocamlopt must not follows -l flags given by *.cm{,x}a and takes them from META files. So from the old world (nocrypto, digestif.0.74 and checkseum.0.0.9), all works. I still need to figure out the new world however (specially about digestif and variants and others provided *.a).

• Compatible with digestif.0.8.0/checkseum.0.9.0
• Compatible with nocrypto

A clear limitation is that for any library which uses a C stub and do not get right information on META files, we will have a link error about some symbols due to -noautolink.

Instead of package providing a library lib_.a and mucking about with META files to force the linker to use the non-standard library name, package will provide lib.a in some target-specific location and something (possibly ocaml-freestanding, but see below) will provide information to mirage needed to set the linker's library search path to use only(!) libraries from the target-specific location(s).

Correct?

Correct at the beginning, not needed now because we use -noautolink. This is the way chosen before and ocamlopt with -output-complete-obj wants to follow flags given by *.cm{,x}a (like zarith.cmxa informs to put -lgmp at the link step).

Possible solution about that:

• -noautolink as I showed in my previous comment
• avoid the link by dune, I still continue to not fully understand the diff between -output-obj and -output-complete-obj and I suspect that the diff is probably the problem - and the main difference between dune and ocamlbuild.

Without pkg-config we lose the ability to compose -L search paths -- what you write above seems to require ocaml-freestanding to know about every possible package in order to be able to emit the -L$(opam config var lib)/. That does not scale, and is the biggest limitation of dropping pkg-config altogether.

-L flags if we look into details was currently given by 2 tools:

• pkg-config for Solo5 stuff (but Xen too)
• ocamlfind for libraries like nocrypto or digestif

About the current status of mirage, we still must have ocaml-freestanding for this purpose. My only update (even if it's little bit hard to follow) is to translate these information to some files (eg. mirage/ocaml-solo5#66) according the dune's format. The old part is available here: https://github.com/mirage/mirage/pull/1020/files#diff-34f96126d5975959270e9225bfd90fa0L29

Now, with dune I firstly moved that to the configure time and, from the request of @samoht, moved this step to the build time.

As an alternative, could we instead have_package_-target install its libraries into a shared directory, such as $(opam config var lib)/mirage-/lib, and this would be added to -L/provided to mirage by some suitable package? I write "some suitable" deliberately as this is probably not ocaml-freestanding's job, but should instead be done by mirage-, i.e. mirage-xen or `mirage-solo5.

I think, you describe in some words what is the current problem. layout of libraries with C stubs to be compatible with MirageOS is not documented. nocrypto has way and digestif has an other way. From this erratic eco-system, we currently want to move all of that to a dune's layout with variants - but still be compatible with old layout.

At the end, the story is only about how to correctly link an unikernel. But I will describe by through in another comment.

Do we need this tool at all? Couldn't ocaml-freestanding or mirage-solo5 generate the information required in a format already suitable for passing to ocamlopt?

We currently need this tool not only for ocaml-freestanding and mirage-solo5 but for libraries like nocrypto and digestif (or bigstringaf). So libraries outside the scope of MirageOS.

Relying on directory precedence only in the presence of multiple libraries with the same name is incredibly fragile and will come back to bite us in the future. Can't we have one and one only specific search path per xen/freestanding/...?

I completely agree with you but the diff between now and what is proposed in this PR is to fix it a bit with what we can have from the eco-system. dune gives an other way but we need to keep in our mind that some packages will not be updated to the dune's layout in a near future.

If you are using ocamlopt to link instead of ld, does that mean you are no longer passing -nostdlib? That would be bad, see above. The unikernel must not accidentally link in host libraries.

ld still is used at the final step for Solo5 and Xen target and we use -nostdlib 👍 . The linker is used by ocamlopt to produce a *.o because dune emits ocamlopt -output-complete-obj ....

Status of link step

So I just would like to clarify a non obvious situation about MirageOS.

Currently, most of problems highlighted by the dunification of MirageOS is about the linking step. This problem is currently fixed by several ways:

• pkg-config
• ocamlfind
• the cmxa format

All of three can give to us information needed to correctly link libraries and C stubs/archives, etc. From what I know, the mostly popular way to get this information is ocamlfind with the META file but it does not fit correctly.

The cmxa format is the default way to get right C stubs/C archives, but it's possibly the wrong way (currently nocrypto.cmxa gives to us -lnocrypto_stubs by default, it's the same for zarith.cmxa).

pkg-config go further and can be more precise about what we needs to properly link our unikernel but we have some concerns about this tool (cc @hannesm).

And, on top of that, we want to move to the dune's variant layout.

Concerns about what mirage does not do

In any way, we decided to let this task to the build system (currently to ocamlbuild/ocamlfind) but I'm not sure that is the right direction to do. MirageOS has a so specific link step and, even if I reach a status when mirage-skeleton is happy with dune, the cost is huge and I'm sure because we clearly use dune to a non-conventional way.

So I start to think about a mirage link to take care about the link step by ourselves with a deal with ocamlopt only. As @mato said, this part is incredibly fragile.

Let's focus about the dunification

I'm not really happy with the status of https://github.com/mirage/mirage.git#clean-dunification-without-pkg-config but the underlying problem is not really about dune but about what the mirage should do on top of the build-system. But I will keep in my mind that and may be provide a better solution for all.

I'm not really happy with the status of https://github.com/mirage/mirage.git#clean-dunification-without-pkg-config

Me neither.

but the underlying problem is not really about dune but about what the mirage should do on top of the build-system. But I will keep in my mind that and may be provide a better solution for all.

We should try and find a proper solution to the problem of "Mirage-compatible" packages, and we should do that now rather than later, since as far as I can tell the current state of the "dunification" just increases complexity and technical debt.

If we can replace pkg-config entirely and get other benefits such as a standardised way of building Mirage-compatible packages, we should consider such a solution "now" (with e.g. a "flag day" for the MirageOS release that includes it), even if that means breaking compatibility with existing packages. Conversely, if the tools (dune, ocamfind, ...) are not up to the task, we should also consider fixing the tools!

Given the complexity involved, the best way to progress on this would be to actually sit down together for a couple of days at a computer, possibly with some dune folks around, and figure it out. However, I don't see that happening this year, I already have too much on my plate with higher-priority work involving building systems that use MirageOS.

Any ideas on how to progress this? @avsm @samoht @hannesm @dinosaure

Attaching today's discussion from Slack, so that we don't lose it.
slack-log.txt