The domain_parallel_spawn_burn tests creates lots of domains that live at the same time, which is causing the CI failure.

You may want to increase the Max_domains value for this test. Max_domains value of 32 should be sufficient as the slots will be reused. See

You can specify the OCAMLRUNPARAM for a single test in the TEST stanza like so:

I was thinking about these pesky programs that change minor heap size programmatically. Maybe they are wrong (the runtime system knows best what the good size is) and should be told to stop doing that, or their requests to change size should be ignored. But assuming we want to keep supporting this idiom, here is one possible way to go about it.

All minor heaps should reside in a range of memory addresses that contain no other data, so that the "is young" test can be implemented as two pointer comparisons. Let's call this range of addresses "the minor area".

Currently, "the minor area" is structured as N blocks of size S, where N is the max number of domains and S the max size for minor heaps, as fixed once and for all (before this PR) or determined when the runtime system starts (this PR). But many other arrangements can be considered, including dynamic allocation within the minor area...

The one parameter that must be determined when the runtime system starts is the total size of the minor area, so that an address range of this size can be reserved. After this, we can have a trivial dynamic allocator that implements malloc and free requests. These caml_minor_area_alloc and caml_minor_area_free functions can then be used to allocate minor heaps when domains are created, and to resize the minor heap for the calling domain exactly like OCaml 4 does it today (do minor GC, free minor heap, allocate new minor heap).

Additional benefit: with a given total minor area size, we can have one or a few domains with a huge minor heap, or a boatload of domains with small minor heaps... No need to decide in advance.

Additional benefit: with a given total minor area size, we can have one or a few domains with a huge minor heap, or a boatload of domains with small minor heaps... No need to decide in advance.

I understand that @sadiqj has experimented with various allocation strategies for the minor heaps that would be more flexible, but it was found to perform noticeably worse than the current approach, for unclear reasons (possibly NUMA effects). See the post-mortem at https://github.com/ocaml-multicore/ocaml-multicore/wiki/Domain-Local-Allocation-Buffers-Addendum

I understand that @sadiqj has experimented with various allocation strategies for the minor heaps that would be more flexible

I forgot about that, thanks for the pointer! However, his schema is more ambitious than mine. With his approach, chunks of the minor area could be acquired by a domain, then released at the next minor GC, then reacquired by another domain. This is a very clever way to address the case of domains having drastically different allocation rates. But I'm not surprised NUMA systems react badly to chunks of memory being "moved" from one core to another this way.

My proposal is much much more modest: every domain still owns a fixed subset of the minor area; it's just that this subset is determined dynamically at domain creation time rather than being partitioned a priori. Maybe that's still well suited to NUMA ?

My proposal is much much more modest: every domain still owns a fixed subset of the minor area; it's just that this subset is determined dynamically at domain creation time rather than being partitioned a priori. Maybe that's still well suited to NUMA ?

If I understand this correctly there may still be an issue after minor heap resizing that a domain ends up with some memory that belonged to a different domain prior to resize and may be on a different core/socket.

One option to avoid this (if we expect resizing minor heaps to be an infrequent operation) might be to munmap and mmap the minor heap?

I also wasn't 100% sold that the DLABs work @gasche mentioned was all NUMA issues, there seemed to be some weird cache effects going on that we didn't understand.

If I understand this correctly there may still be an issue after minor heap resizing that a domain ends up with some memory that belonged to a different domain prior to resize and may be on a different core/socket.

I would expect minor heap resizing to occur very rarely (like, once at the beginning of the program), so this may not be much of a concern.

On the other hand, unmapping then remapping could be a good opportunity to change mapping flags and e.g. force the memory to be allocated for good? (I'm probably talking nonsense here).

On the third hand, I'm not sure that unmapping then remapping always succeeds...

caml_minor_area_alloc and caml_minor_area_free functions can then be used to allocate minor heaps when domains are created, and to resize the minor heap for the calling domain exactly like OCaml 4 does it today (do minor GC, free minor heap, allocate new minor heap).

One concern I have with such a dynamic scheme is that we may observe non-deterministic domain creation failures. For example, consider for the sake of argument that we create a 6 MB minor heap area to be shared among all of the domains. Let us assume that each domain starts with a default 2 MB minor heap. Consider the following program:

let main () =
  let d1 = Domain.spawn (fun _ -> 
    Gc.set { (Gc.get()) with Gc.minor_heap_size = 4194304 (* 4M *) })
  in
  let d2 = Domain.spawn (fun _ -> ()) in
  Domain.join d1;
  Domain.join d2

The spawn of d2 may fail non-deterministically depending on when d1 gets to resize the minor heap. If d2 is spawned before d1 gets to resize, then the resize request will fail. I suspect that it may end up being difficult to debug the program if many domains enter and leave.

This PR proposes a simpler API. You can have Max_domains number of domains running simultaneously. If domain creation fails, then the error message directs the user to look at this option. We may consider doing something similar for resize request. Currently, it is a fatal error if the resize request fails, but we can direct the user to look at the maximum minor heap size OCAMLRUNPARAM option.

It is a bit unsatisfying that Coq will fatal error if the default Minor_heap_max is smaller than 256 MB in this PR. Coq will have to be run with OCAMLRUNPARAM="N=32M". Perhaps option number 2

Silently bump N to be at least as much as s. Choose the default values such as N >= s.

is not so bad with N=32M and s=256k.

Overall, I feel that this PR is proposing useful improvements

• No fixed limit for minor heap size and maximum domains
• Valgrind can work

I am keen to get to a state where the PR is forwards compatible with the experiments that we would like to do on alternate minor heap designs, and provides a simple API to the users.

The one parameter that must be determined when the runtime system starts is the total size of the minor area, so that an address range of this size can be reserved.

Somewhat naive question: why does the address range need to be reserved once and for all at the beginning? I would have thought that a new larger minor area could be mapped dynamically as the required size grows, doing a minor collection when it is, since there aren't any pointers into the minor area that survive a minor collection.

Somewhat naive question: why does the address range need to be reserved once and for all at the beginning? I would have thought that a new larger minor area could be mapped dynamically as the required size grows, doing a minor collection when it is, since there aren't any pointers into the minor area that survive a minor collection.

I agree. Resize is a rare operation, and its performance is not a particular concern. On a resize request, we may perform a minor collection, and then in a stop-the-world section munmap the previous minor heap area, and reserve a larger area. Then the first thing each domain does is to commit the memory for its minor heap, and proceed with mutator execution. This will ensure the same locality benefits that we have currently.

Potentially, the same idea can work for the case when there are no free domain slots during a domain spawn.

On a resize request, we may perform a minor collection, and then in a stop-the-world section munmap the previous minor heap area, and reserve a larger area. Then the first thing each domain does is to commit the memory for its minor heap, and proceed with mutator execution. This will ensure the same locality benefits that we have currently.

Agree. This was my suggestion earlier up in the thread, we don't necessarily need to keep the same minor heap range as long as it's done during the stw after the minor heap has been collected.

It is a bit unsatisfying that Coq will fatal error if the default Minor_heap_max is smaller than 256 MB in this PR. Coq will have to be run with OCAMLRUNPARAM="N=32M"

This is not acceptable. Tweaking OCAMLRUNPARAM is OK for developers but not for end-users. (Plus, Windows users don't even know how to set an environment variable.) Standalone applications such as Coq or CompCert must be able to set their run-time parameters (minor heap size, max number of domains, etc) programmatically, at the beginning of execution. It's OK if it can only be done before the first domain is spawned.

Several ways to do this have been proposed in this PR. Why don't we agree first on functionality, then on a design, and only then submit and review PRs ?

Clearly, it is unacceptable to break existing programs by requiring an environment variable to be set for them.

What if we keep the default Minor_heap_max at the original value of 2GB, so that existing programs are not affected (then, developers who need to run Valgrind or AFL can use the OCAMLRUNPARAM to lower this limit)?

It has become clear during the discussion that, moving forward, there is potential in exploring a better solution for giving the programmer control over the virtual memory reservations performed by the runtime.

I'm closing this in favor of continuing the discussion on #10971.

Thanks for the clarification @shindere.