Skip to content

Add some dune files#2093

Merged
trefis merged 11 commits intoocaml:trunkfrom
trefis:dune
Oct 10, 2018
Merged

Add some dune files#2093
trefis merged 11 commits intoocaml:trunkfrom
trefis:dune

Conversation

@trefis
Copy link
Copy Markdown
Contributor

@trefis trefis commented Oct 5, 2018

Steps:

  • merlin is not usable when working on trunk (different cmis ⇒ segfault)
  • we would get cmis that merlin understand if we built the compiler with a preinstalled one instead
  • "Oh dear God, I don't want to touch the makefiles"
  • add some dune files
  • $ dune build -w @world

The last command currently builds bytecode versions of ocamlc, ocamlopt, ocamldebug, ocamldoc, ocamltest, the toplevel.
The idea is that dune can be used for development (that is why we only build bytecode executables), to get fast feedback and a working merlin environment.
It is not meant to replace the usual build process.


I extracted a couple of things from the makefiles so they can be more easily reused by dune. I'm guessing @shindere will probably want to have a look.


It probably doesn't work on windows, I'm sure @dra27 will send a follow-up PR.

@Armael
Copy link
Copy Markdown
Member

Armael commented Oct 6, 2018

Am using this right now, it is quite convenient. A few issues/remarks:

  • Some stuff is failing it seems:
Uncaught exception: Sys_error("topstart.exe: No such file or directory")
  • It would be nice to have a target equivalent to make runtop
  • You seem to be missing some warnings-as-errors that are enabled with the standard makefile

(with-stdout-to %{targets}
(progn
(bash "echo \\# 1 \\\"`cat contains-input-name`\\\"")
(bash "%{dep:../tools/cvt_emit.exe} < `cat contains-input-name`"))))))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another way to do this would be to modify cvt_emit to take a filename and include a line directive in the output.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I agree. But I tried to minimize the changes in this PR, to limit the amount of review required. I'll change that in a follow-up PR I think.

(rule
(targets ocaml.byte)
(action (run %{ocaml_where}/expunge topstart.exe %{targets}
; FIXME: inlined $(STDLIB_MODULES) ... minus Labels ones ...
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed with @trefis, we have some code in utop to avoid writing the list of modules explicitly:

https://github.com/ocaml-community/utop/blob/master/src/top/expunge/expunge.ml

Called as follow: https://github.com/ocaml-community/utop/blob/master/src/top/expunge/expunge.ml

We could modify expunge to do that directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to do that in a follow-up PR.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 8, 2018

Some stuff is failing it seems:

Indeed, that's me doing some last minute cleanups of the dune files without testing properly. It should now be fixed.

It would be nice to have a target equivalent to make runtop

We've discussed this on another medium, but to answer publicly: indeed it would, and it would also be nice to be able to run the testsuite for example.
But it's not possible at the moment. In particular for the toplevel: there's no stdlib available, the stdlib was built with a different version of ocaml than the toplevel is at, so it's not able to load the .cmi files.

We might have some sort of bootstrap procedure with dune in the future (based on workspaces?), but until then, dune is really only usable for building things, not for actual testing.

You seem to be missing some warnings-as-errors that are enabled with the standard makefile

Fair point, I pushed a commit fixing this.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 8, 2018

FTR check-typo is currently unhappy because there are no header in the dune files. Should I add some? Could I opt-out dune files of that check?

@gasche
Copy link
Copy Markdown
Member

gasche commented Oct 8, 2018

Makefiles have headers, so why not Dune files?

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 8, 2018

Well, the .merlin file didn't have a header. I considered dune files to be in the same scope as the .merlin was, that is: convenient for developers, but unessential.

But you're right: let's look towards the (unavoidable) future where dune finally takes over. I'll add some headers.

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 9, 2018

I don't understand the title of this PR, perhaps because I am missing a
cultural reference? Would you please mind changing the title to something that
would describe this PR's content in a more factual way?

I am a bit puzzled by the removal of the .merlin file because that will
impact those developers who do not plane to switch to Dune right now.
Would it be possible that this PR does not touch the .merlin file?
Is merlin really broken on trunk?

Was it on purpose that you kept the mention of Inria in the headers of the
Dune file? I am wondering whether this is correct? Perhaps @xavierleroy or
@damiendoligez will know?

I think bytecomp/generate_runtimedef.sh and runtime/gen_primitives.sh
should have headers, too.

Same for utils/Makefile and the comment "The configuration file" is not
precise enough, IMO. This file generates the configuration file. It would
also be nice if you could make it use ROOTDIR as the other sub-makefiles
do, although I know it's not strictly necessary, just for sake of
consistency. Also, you should remove the BYTERUN variable that has been
removed in trunk recently, I guess this comes frm a rebase.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 10, 2018

The title was indeed a reference to dune. But sure, I'll change it.

I am a bit puzzled by the removal of the .merlin file because that will
impact those developers who do not plane to switch to Dune right now.
Would it be possible that this PR does not touch the .merlin file?
Is merlin really broken on trunk?

Merlin is completely unusable on trunk right now, and was also completely unusable for a few weeks/months before 4.07.0 was released.
Before that, it used to work partially (i.e. some features worked fine, some others which require cmt files just didn't work at all), but that seems to have become the exception rather than the rule.
What this PR gives you, provided you're willing to install dune, is a fully working merlin.

And no, we can't keep the the .merlin file commited.

Was it on purpose that you kept the mention of Inria in the headers of the Dune file?

Yes, but it might be a mistake. I have no idea.

I think bytecomp/generate_runtimedef.sh and runtime/gen_primitives.sh should have headers, too.

Indeed! I'll push a fix.

the comment "The configuration file" is not precise enough, IMO.

Fair enough. I had just copied it with the actual rule from the main Makefile. But I'll edit it.

It would also be nice if you could make it use ROOTDIR as the other sub-makefiles do, although I know it's not strictly necessary, just for sake of consistency.

Sure!

Also, you should remove the BYTERUN variable that has been removed in trunk recently, I guess this comes frm a rebase.

You guessed right. Will do.


Thanks for the review @shindere !

(library
(name str)
(modes byte)
(wrapped false)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need wrapped to be false? There is only one str module in the library currently.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it doesn't!
I was on auto-pilot at this point :p

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 10, 2018 via email

@trefis trefis changed the title Add some spice Add some dune files Oct 10, 2018
@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 10, 2018

I pushed fixes for all of your comments!

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 10, 2018

I think all the headers are now correct, and the CI is happy.
Are people OK if I (squash and) merge this now?

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 10, 2018 via email

@trefis trefis merged commit d68e0e2 into ocaml:trunk Oct 10, 2018
@trefis trefis deleted the dune branch October 10, 2018 15:16
@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 10, 2018

Thank you again for the review @shindere!

I set a more descriptive name (reusing your suggestion) for the commit, I don't think the PR itself needs anything more descriptive now that the discussion is over.

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 10, 2018 via email

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 11, 2018

I'll have a look now.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 11, 2018

@shindere I managed to reproduce the bootstrap failure without my commit:

$ git reset --hard origin/trunk
$ make disclean
$ git clean -dfx
$ git reset --hard HEAD^
$ make world
$ sed -i 's/\x23define \+EXEC_MAGIC \+\x22Caml1999X023\x22/#define EXEC_MAGIC "toto"/' runtime/caml/exec.h
$ sed -i 's/let \+exec_magic_number \+= \+\x22Caml1999X023\x22/let exec_magic_number = "toto"/' utils/config.mlp
$ patch -p1 < tools/ci/inria/remove-sinh-primitive.patch
$ make coreall
$ make bootstrap
../boot/ocamlrun ../boot/ocamlc -g -nostdlib -I ../boot -use-prims ../runtime/primitives -I .. make_opcodes.ml -o make_opcodes
make[2]: Leaving directory `/usr/local/home/trefis/ocaml/trunk/tools'
runtime/ocamlrun tools/make_opcodes -opcodes < runtime/caml/instruct.h > bytecomp/opcodes.ml
Fatal error: the file 'tools/make_opcodes' has not the right magic number: expected toto, got 
make[1]: *** [bytecomp/opcodes.ml] Error 2

I wanted to look further, but I am absolutely incapable of decyphering the content of tools/Makefile.

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 11, 2018 via email

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Oct 11, 2018

You didn't ./configure ?

I did, you can't build anything if you don't ./configure.
Sorry for the omission.

The magic number is expected to be 12 characters

Ah! The error indeed disappears when I use a magic number of the right length. Fantastic.

I'll keep digging, thank you @shindere!

@shindere
Copy link
Copy Markdown
Contributor

shindere commented Oct 11, 2018 via email

@ahh
Copy link
Copy Markdown
Contributor

ahh commented Apr 13, 2019

@trefis Two questions:

(a) At current head of trunk, dune build @world is giving me a dozen-ish errors of the form:

      ocamlc .ocamlmiddleend.objs/byte/inlining_cost.{cmo,cmt} (exit 2)
(cd _build/default && /home/ahh/.opam/ocaml-base-compiler/bin/ocamlc.opt -w @a-4-29-40-41-42-44-45-48-58-59-60-40 -strict-sequence -strict-formats -short-paths -keep-locs -w +a-4-9-41-42-44-45-48 -principal -nostdlib -g -bin-annot -I .ocamlmiddleend.objs/byte -I .ocamlcommon.objs/byte -I .ocamlcommon.objs/native -I stdlib/.stdlib.objs/byte -I stdlib/.stdlib.objs/native -intf-suffix .ml -no-alias-deps -opaque -o .ocamlmiddleend.objs/byte/inlining_cost.cmo -c -impl inlining_cost.ml)
File "middle_end/flambda/inlining_cost.ml", line 572, characters 25-31:
Error: This expression has type CamlinternalFormatBasics.float_kind_conv
       but an expression was expected of type
         CamlinternalFormatBasics.float_conv =
           CamlinternalFormatBasics.float_flag_conv *
           CamlinternalFormatBasics.float_kind_conv

I believe Dune is using my OPAM ocaml 4.07.01, though it's hard for me to be certain (I'm not sure how to get Dune to tell me. EDIT: I just realized this is almost certainly what I'm using as it gives the command line above, woops!) Is this a KI? Should dune build @world succeed?

(b) Related: I was getting a bunch more errors before, when I without realizing it had a old 4.05 install of Ocaml from OPAM. Dune gave me no warning. Is there a way to change the dune file to specify a required compiler version?

@gasche
Copy link
Copy Markdown
Member

gasche commented Apr 13, 2019

The first is a known issue with the dune build: because it uses your current OCaml install, and not the bootstrapped compiler of the compiler distribution, the build will break whenever the two are not compatible (during the development, some changes create incompatibilities, changing the type of format in certain ways is one of them). With the current way the dune build system for the compiler works, you can just assume that the build of trunk will remain broken in trunk (not 4.08) until OCaml 4.09 is released. In the meantime (before 4.09 gets released or the dune build changes to use the bootstrap compiler), using the makefiles works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants