Note: I wonder if there is a deeper problem here.

1. Can OCaml 5.0 users use caml_atfork_hook, is it exposed in the public API? (And, ideally, documented somewhere?)
2. Is it possible to write fork()-using C code that works with both 4.x and 5.0?

Maybe we can recommend to always use unix_fork from C code, and never the naked fork() call, to avoid this issue?

(Note: even with 4.x, there were reasons to recommend this, for example for Eventlog users, as unix_fork has the right cleanup logic there.)

It's not obvious to me why the child process uses the OCaml runtime. If we were to implement spawning using posix_spawn instead of fork + exec (resulting in nice performance gains), there would be no opportunity for the child to use the OCaml runtime. For the same reason, the Win32 implementation of command spawning cannot use the OCaml runtime. So, what's going on here?

The ocamltest implementation of "running a command" takes as parameter an OCaml channel for logging. At the point where the child sets things up to call excevp, any error will call error-reporting functions that write to this logging channel by calling OCaml runtime functions. See run_command_child and logToChannel.

In the medium term, maybe we could reconsider the idea of maintaining yet another implementation of spawn. In the short term, the PR is a minimal fix for a bug introduced by the change of runtime assumptions around fork().

At the point where the child sets things up to call excevp, any error will call error-reporting functions that write to this logging channel by calling OCaml runtime functions

Well, I find this questionable, even in OCaml 4. For example, there's a risk of double-flushing of OCaml channels (once in the parent, once in the child). resulting in duplicate output.

Good point. I did wonder about flushing before the fork() and then writing to the file descriptor directly, for example. But this also requires subtle and invasive changes to the implementation. I'm not currently planning to reimplement the run_command routine of ocamltest.

On the other hand, I think we do have OCaml programs that fork() and then use the OCaml runtime on both sides. I would expect those programs to call Unix.fork directly, but some could also possibly fork from C. We have decided that forking a multi-domain program would not be supported (and Unix.fork will error in this situation), but we intend to support forking a single-domain program, even under the Multicore runtime. So we can take this one example from the compiler distribution as a case study of compatibility issues that our users could encounter inthe wild. (Possibly as well in situations where the fork() usage is questionable, but that work fine in practice under 4.x.)

On the other hand, I think we do have OCaml programs that fork() and then use the OCaml runtime on both sides. I would expect those programs to call Unix.fork directly, but some could also possibly fork from C.

I really hope this pattern (fork in C, then use the OCaml runtime system in the child process) is not to be found in the wild. If I understand correctly, this pattern causes a fatal error in 5.0, so we'll hear about it.

Thanks! I updated and rebased.

If I understand correctly, this pattern causes a fatal error in 5.0, so we'll hear about it.

It fails on my machine on the first time that we try to release the runtime lock / enter a blocking section. (I'm slightly nervous at the idea that our options to fix the issue in a backward-compatible ways will be very limited once 5.0 and 4.14 are release without a coherent support for forking, but oh well, let's assume that nobody does that indeed.)