Skip to content

Ci use mirage4#166

Merged
hannesm merged 6 commits intomirage:mainfrom
hannesm:ci-use-mirage4
Nov 7, 2022
Merged

Ci use mirage4#166
hannesm merged 6 commits intomirage:mainfrom
hannesm:ci-use-mirage4

Conversation

@hannesm
Copy link
Copy Markdown
Member

@hannesm hannesm commented Sep 13, 2022

//cc @TheLortex applying your suggestion from mirage/mirage#1346 -- but somehow the CI fails since now a mirage build attempts to cross-compile everything, and not only the unikernel.

Also, in the generated Makefile (at top level) the build target omits the -f mirage/config.ml -- is this intentional?

@TheLortex
Copy link
Copy Markdown
Member

Currently you have to use -f mirage/config.ml everywhere, which I agree is not very practical.

Also, in the generated Makefile (at top level) the build target omits the -f mirage/config.ml -- is this intentional?

That's a bug indeed.

@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Sep 20, 2022

Thanks @TheLortex -- though do you have advise how to fix the CI here?

The failure I see at https://cirrus-ci.com/task/6202563144974336?logs=test_mirage#L55 is duplicated below -- it looks like the mirage build -f mirage/config.ml attempts to build all the sources and fails since there are packages (eio, async, ...) that are not relevant for the unikernel, but part of the source tree...

+ gmake depend
using overlay repository mirage: [opam-overlays, mirage-overlays] 
[WARNING] Running as root is not recommended
[opam-overlays] Initialised
[NOTE] Repository opam-overlays has been added to the selections of switch 4.13.1 only.
       Run `opam repository add opam-overlays --all-switches|--set-default' to use it in all existing switches, or in newly created switches, respectively.
[WARNING] Running as root is not recommended
[mirage-overlays] Initialised
[NOTE] Repository mirage-overlays has been added to the selections of switch 4.13.1 only.
       Run `opam repository add mirage-overlays --all-switches|--set-default' to use it in all existing switches, or in newly created switches, respectively.
 ↳ generate lockfile for monorepo dependencies
==> Using 1 locally scanned package as the target.
==> Found 74 opam dependencies for the target package.
==> Querying opam database for their metadata and Dune compatibility.
==> Calculating exact pins for each of them.
==> Wrote lockfile with 38 entries to mirage/mirage/crypto-test-unix.opam.locked. You can now run opam monorepo pull to fetch their sources.
removing overlay repository [opam-overlays, mirage-overlays]
[WARNING] Running as root is not recommended
Repositories removed from the selections of switch 4.13.1. Use '--all' to forget about them altogether.
[WARNING] Running as root is not recommended
Repositories removed from the selections of switch 4.13.1. Use '--all' to forget about them altogether.
 ↳ opam install switch dependencies
[WARNING] Running as root is not recommended
Nothing to do.
 ↳ install external dependencies for monorepo
==> Using lockfile mirage/mirage/crypto-test-unix.opam.locked
 ↳ fetch monorepo rependencies in the duniverse folder
==> Using lockfile mirage/mirage/crypto-test-unix.opam.locked
Successfully pulled 38/38 repositories
+ mirage build -f mirage/config.ml
File "rng/eio/dune", line 4, characters 12-15:
4 |  (libraries eio cstruct logs mirage-crypto-rng duration mtime.clock.os))
                ^^^
Error: Library "eio" not found.
-> required by library "mirage-crypto-rng-eio" in _build/default/rng/eio
-> required by _build/default/META.mirage-crypto-rng-eio
-> required by alias all
-> required by alias default
File "mirage/dune.build", line 19, characters 16-31:
19 |  (enabled_if (= %{context_name} "default"))
                     ^^^^^^^^^^^^^^^
Error: %{context_name} is only available since version 2.7 of the dune
language. Please update your dune-project file to have (lang dune 2.7).
File "tests/dune", line 76, characters 43-51:
76 |  (libraries mirage-crypto-rng-eio duration eio_main)
                                                ^^^^^^^^
Error: Library "eio_main" not found.
-> required by _build/default/tests/test_eio_entropy_collection.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "astring" in _build/default/mirage/duniverse/astring/src
- "astring" in /.opam/4.13.1/lib/astring
  -> required by library "alcotest.stdlib_ext" in
     /.opam/4.13.1/lib/alcotest/stdlib_ext
  -> required by library "alcotest.engine" in
     /.opam/4.13.1/lib/alcotest/engine
  -> required by library "alcotest" in /.opam/4.13.1/lib/alcotest
-> required by executable test_ec in tests/dune:53
-> required by _build/default/tests/test_ec.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "base" in _build/default/mirage/duniverse/base/src
- "base" in /.opam/4.13.1/lib/base
  -> required by library "core" in /.opam/4.13.1/lib/core
  -> required by library "async_kernel.config" in
     /.opam/4.13.1/lib/async_kernel/config
  -> required by library "async_kernel" in /.opam/4.13.1/lib/async_kernel
  -> required by library "async.async_command" in
     /.opam/4.13.1/lib/async/async_command
  -> required by library "async" in /.opam/4.13.1/lib/async
  -> required by library "mirage-crypto-rng-async" in
     _build/default/rng/async
-> required by executable test_entropy_collection_async in tests/dune:41
-> required by _build/default/tests/test_entropy_collection_async.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "stdlib-shims" in _build/default/mirage/duniverse/stdlib-shims/src
- "stdlib-shims" in /.opam/4.13.1/lib/stdlib-shims
  -> required by library "ounit2.advanced" in
     /.opam/4.13.1/lib/ounit2/advanced
  -> required by library "ounit2" in /.opam/4.13.1/lib/ounit2
  -> required by library "test_common" in _build/default/tests
-> required by executable test_pk_runner in tests/dune:27
-> required by _build/default/tests/test_pk_runner.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "base" in _build/default/mirage/duniverse/base/src
- "base" in /.opam/4.13.1/lib/base
  -> required by library "core" in /.opam/4.13.1/lib/core
  -> required by library "async_kernel.config" in
     /.opam/4.13.1/lib/async_kernel/config
  -> required by library "async_kernel" in /.opam/4.13.1/lib/async_kernel
  -> required by library "async.async_command" in
     /.opam/4.13.1/lib/async/async_command
  -> required by library "async" in /.opam/4.13.1/lib/async
-> required by
   _build/default/rng/async/.mirage_crypto_rng_async.objs/byte/mirage_crypto_rng_async.cmo
-> required by _build/default/rng/async/mirage_crypto_rng_async.cma
-> required by alias rng/async/all
-> required by alias default
Error: Conflict between the following libraries:
- "cstruct" in _build/default/mirage/duniverse/ocaml-cstruct/lib
- "cstruct" in /.opam/4.13.1/lib/cstruct
  -> required by library "hex" in /.opam/4.13.1/lib/hex
-> required by
   _build/default/tests/wycheproof/.wycheproof.objs/byte/wycheproof.cmo
-> required by _build/default/tests/wycheproof/wycheproof.cma
-> required by alias tests/wycheproof/all
-> required by alias default
Error: Conflict between the following libraries:
- "stdlib-shims" in _build/default/mirage/duniverse/stdlib-shims/src
- "stdlib-shims" in /.opam/4.13.1/lib/stdlib-shims
  -> required by library "ounit2.advanced" in
     /.opam/4.13.1/lib/ounit2/advanced
  -> required by library "ounit2" in /.opam/4.13.1/lib/ounit2
-> required by _build/default/tests/.test_common.objs/byte/test_common.cmo
-> required by _build/default/tests/test_common.cma
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "cstruct" in _build/default/mirage/duniverse/ocaml-cstruct/lib
- "cstruct" in /.opam/4.13.1/lib/cstruct
  -> required by library "randomconv" in /.opam/4.13.1/lib/randomconv
-> required by
   _build/default/tests/.test_common_random.objs/byte/test_common_random.cmo
-> required by _build/default/tests/test_common_random.cma
-> required by alias tests/all
-> required by alias default
(cd _build/default/mirage/duniverse/ocaml-gmp/src && /usr/local/bin/opam config subst build.sh)
[WARNING] Running as root is not recommended
(cd _build/default/src-freestanding && /usr/local/bin/gmake)
touch libmirage_crypto_freestanding_stubs.a
(cd _build/default/ec-freestanding && /usr/local/bin/gmake)
touch libmirage_crypto_ec_freestanding_stubs.a
(cd _build/default/mirage/duniverse/Zarith && /usr/local/bin/bash -e -u -o pipefail -c 'env CC="cc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC" LDFLAGS="-L/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" CFLAGS="-I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" CPPFLAGS="-I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" ./configure --ocamllibdir /.opam/4.13.1/lib/ocaml')
binary ocaml: found in /.opam/4.13.1/bin
binary ocamlc: found in /.opam/4.13.1/bin
binary ocamldep: found in /.opam/4.13.1/bin
binary ocamlmklib: found in /.opam/4.13.1/bin
binary ocamldoc: found in /.opam/4.13.1/bin
binary cc: found in /usr/bin
binary ocamlopt: found in /.opam/4.13.1/bin
checking compilation with cc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/: working
include caml/mlvalues.h: found
library dynlink.cmxa: found
binary ocamlfind: found in /.opam/4.13.1/bin
OCaml's word size is 64
binary uname: found in /usr/bin
include gmp.h: found
library gmp: found
OCaml supports -bin-annot to produce documentation
detected configuration:
  native-code:          yes
  dynamic linking:      yes
  defines:              -DHAS_GMP 
  libraries:            -L/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/ -lgmp
  C options:            -I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/
  installation path:    /.opam/4.13.1/lib
  installation method   findlib
configuration successful!
now type "make" to build
then type "make install" or "sudo make install" to install
+ exit 1

@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Nov 7, 2022

@TheLortex do you have an idea how to solve this issue, and get Mirage4 to be used in the CI with the example unikernel?

I guess it boils down to "how to compile the example unikernel in here with the HEAD branch of mirage-crypto-*?"

@TheLortex
Copy link
Copy Markdown
Member

Hi, the mirage build command should only ask dune to build the unikernel. Currently it's invoking dune build --root . which is asking dune to build everything in the workspace, so that's not correct. The invocation of dune to only build the unikernel would be dune build --root . mirage/dist/

@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Nov 7, 2022

Thanks again @TheLortex, I managed to adapt the .test-mirage.sh to use the dune build --root . mirage/dist/ directly instead of a mirage build, and the CI is passing now \o/

hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 5, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 5, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 5, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 9, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
  This means, a "mirage-crypto-rng.lwt" should now be "mirage-crypto-rng-lwt"
  in your dune file (or in META requires, or in _tags).
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
  This means any "Mirage_crypto.AES.CCM" should now be "Mirage_crypto.AES.CCM16"
  and any "CCM.of_secret ~maclen:16 key" should now be "CCM16.of_secret key"
  Any occurrence of "Mirage_crypto.Cipher_block.S.CCM" should now be
  "Mirage_crypto.Cipher_block.S.CCM16"
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
  This means:
  - "Mirage_crypto_rng_lwt.initialize ()" should now be
    "Mirage_crypto_rng_lwt.initialize (module Mirage_crypto_rng.Fortuna)"
  - "Mirage_crypto_rng_unix.initialize ()" should now be
    "Mirage_crypto_rng_unix.initialize (module Mirage_crypto_rng.Fortuna)"
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 11, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
  This means, a "mirage-crypto-rng.lwt" should now be "mirage-crypto-rng-lwt"
  in your dune file (or in META requires, or in _tags).
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
  This means any "Mirage_crypto.AES.CCM" should now be "Mirage_crypto.AES.CCM16"
  and any "CCM.of_secret ~maclen:16 key" should now be "CCM16.of_secret key"
  Any occurrence of "Mirage_crypto.Cipher_block.S.CCM" should now be
  "Mirage_crypto.Cipher_block.S.CCM16"
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
  This means:
  - "Mirage_crypto_rng_lwt.initialize ()" should now be
    "Mirage_crypto_rng_lwt.initialize (module Mirage_crypto_rng.Fortuna)"
  - "Mirage_crypto_rng_unix.initialize ()" should now be
    "Mirage_crypto_rng_unix.initialize (module Mirage_crypto_rng.Fortuna)"
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
hannesm added a commit to hannesm/opam-repository that referenced this pull request Feb 13, 2023
…age, mirage-crypto-rng-lwt, mirage-crypto-rng-eio, mirage-crypto-rng-async, mirage-crypto-pk and mirage-crypto-ec (0.11.0)

CHANGES:

- BREAKING split mirage-crypto-rng-lwt away from mirage-crypto-rng (mirage/mirage-crypto#168
  @hannesm, reported by @bikallem mirage/mirage-crypto#158)
  This means, a "mirage-crypto-rng.lwt" should now be "mirage-crypto-rng-lwt"
  in your dune file (or in META requires, or in _tags).
- AEAD API improvements: provide tag_size, of_secret, and functions that deal
  with the tag separately (mirage/mirage-crypto#171 @hannesm, fixes mirage/mirage-crypto#74 mirage/mirage-crypto#144 @orbitz @anmonteiro)
  Only CCM16 (with tag size 16) is now exposed, the former API does not exist
  anymore (passing `~maclen` to `of_secret`), according to sherlocode the only
  usage was CCM16 anyways
  This means any "Mirage_crypto.AES.CCM" should now be "Mirage_crypto.AES.CCM16"
  and any "CCM.of_secret ~maclen:16 key" should now be "CCM16.of_secret key"
  Any occurrence of "Mirage_crypto.Cipher_block.S.CCM" should now be
  "Mirage_crypto.Cipher_block.S.CCM16"
- BREAKING unify RNG initialization (reported by @talex5 in mirage/mirage-crypto#155, fixes mirage/mirage-crypto#160,
  PR mirage/mirage-crypto#162 @hannesm)
  This means:
  - "Mirage_crypto_rng_lwt.initialize ()" should now be
    "Mirage_crypto_rng_lwt.initialize (module Mirage_crypto_rng.Fortuna)"
  - "Mirage_crypto_rng_unix.initialize ()" should now be
    "Mirage_crypto_rng_unix.initialize (module Mirage_crypto_rng.Fortuna)"
- remove mirage 3 cross-compilation runes (mirage/mirage-crypto#163 @hannesm)
- CI: mirage-crypto-rng-eio requires ocaml 5 and dune 2.7 (mirage/mirage-crypto#170 @hannesm, fixes
  mirage/mirage-crypto#169 thanks to @bikallem @talex5)
- CI: use miage 4 (mirage/mirage-crypto#166 @hannesm)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants