Skip to content

Compile tools/make_opcodes with -custom#10293

Closed
dra27 wants to merge 2 commits intoocaml:trunkfrom
dra27:bootstrap-make_opcodes
Closed

Compile tools/make_opcodes with -custom#10293
dra27 wants to merge 2 commits intoocaml:trunkfrom
dra27:bootstrap-make_opcodes

Conversation

@dra27
Copy link
Copy Markdown
Member

@dra27 dra27 commented Mar 12, 2021

Prior to #803, bytecomp/opcodes.ml* and tools/opnames.ml were generated by awk and sed scripts. They're now generated infinitely less hideously, but more bootstrappingly complicatedly, with an ocamllex script.

@ctk21 hit a problem with multicore trying to change bytecode opcodes (the aim is in fact to remove opcodes which are no longer required, but which were not originally added at the end of the opcode list). Presently, the build system assumes that tools/make_opcodes executes with runtime/ocamlrun but this is incorrect during a change to the opcodes. After make core has been run and changes applied, tools/make_opcodes should be executed with boot/ocamlrun, not runtime/ocamlrun. That change is easy enough to make, but the bootstrap cycle is more complicated - the correct runtime gets cleaned during the bootstrap.

This PR is the simplest path for fixing this - we store the runtime used to build tools/make_opcodes with tools/make_opcodes by using -custom! I don't think there's any downside to this, beyond slightly more space being used during the build - we don't install the file, so none of the usual problems with -custom apply. I don't think we were ever planning on completely removing -custom, but if we did then it would be perfectly reasonable to switch to -output-complete-exe, I haven't done that here because -custom is technically quicker (it's a copy vs calling the linker).

Note that #1659 would not have been affected by this because an added opcode means that tools/make_opcodes was executable at all times with runtime/ocamlrun. However, the issue would be seen if the exec magic number were bumped at the same time (it wasn't in that PR, although arguably it ought to have been).

dra27 added 2 commits March 12, 2021 12:27
Ensures that it's always executable, even if the bytecode interpreter is
being bootstrapped with changed opcodes.
The -custom option will use whatever configuration built boot/ocamlc,
which may well not be correct! Instead, use the cat'ing trick directly.
@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Mar 12, 2021

Oops, I forgot that -custom always builds a runtime, even if not needed. This is still technically "custom"-runtime, but not using -custom, since that will call the linker with whatever boot/ocamlc was configured with.

@xavierleroy
Copy link
Copy Markdown
Contributor

I agree we have a problem, but I don't think -custom is the solution. It's a bit of a hack and I would very much like to keep it out of the bootstrap path.

I see in the toplevel Makefile that make_opcodes is run with runtime/ocamlrun. Bootstrap-wise, it would work better with boot/ocamlrun, provided that the tools is not built with -use-prims $(ROOTDIR)/runtime/primitives (see tools/Makefile).

I think we need to check tools/ w.r.t. bootstrapping and understand what we want first.

@xavierleroy
Copy link
Copy Markdown
Contributor

Also: insert my standard rant about using OCaml too early while building OCaml. We keep running into this mistake like a moth keeps flying into a bright light.

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Mar 12, 2021

I agree we have a problem, but I don't think -custom is the solution. It's a bit of a hack and I would very much like to keep it out of the bootstrap path.

I see in the toplevel Makefile that make_opcodes is run with runtime/ocamlrun. Bootstrap-wise, it would work better with boot/ocamlrun, provided that the tools is not built with -use-prims $(ROOTDIR)/runtime/primitives (see tools/Makefile).

I think we need to check tools/ w.r.t. bootstrapping and understand what we want first.

I did some other poking around before settling on -custom. If we compile tools/make_opcodes without -use-prims and switch over to running it with boot/ocamlrun, we need to do something different at the first partialclean in coreboot which at present erases tools/make_opcodes. We just did promote-cross, so that means we end up building tools/make_opcodes for the new runtime but then try to run it on the old runtime which is still in boot/ocamlrun. It works for me, I just considered it was more invasive than using the cat trick for ocamlrun.

Also: insert my standard rant about using OCaml too early while building OCaml. We keep running into this mistake like a moth keeps flying into a bright light.

I agree, although let's not forget the amount of fighting we also do with sed and awk! Another option, a la #1078, might be to add the script as an option in boot/ocamlc?

@xavierleroy
Copy link
Copy Markdown
Contributor

let's not forget the amount of fighting we also do with sed and awk

We're in 2021 and there are better scripting languages widely available and used. E.g. Python.

@damiendoligez
Copy link
Copy Markdown
Member

Maybe we could decide that the problem is not worth solving. Do we really need to remove opcodes from the list? Let's just turn them into NOPs and declare them unused/reserved/deprecated/whatever.

@dra27
Copy link
Copy Markdown
Member Author

dra27 commented Mar 15, 2021

For this particular instance that wouldn't work, because the aim was to bring the opcode values back into line with trunk. It seems safe to assume that it won't happen often, but that it'll never happen again - in this case, documenting it almost seems more long-winded than fixing it?

@gasche
Copy link
Copy Markdown
Member

gasche commented Apr 18, 2021

My understanding is that people agree that there is a problem worth solving (at least I would agree), but the specific solution proposed in this PR has been clearly vetoed: no -custom in the bootstrap path.

I'm feeling bold today on my closing-PR-spree, so let me just close this. @dra27, please of course feel free to revisit with another option (you mentioned obscure changes to coreboot, or alternatively embedding more logic in ocamlc), to keep discussing them here if useful, and to reopen if that was a bit too bold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants