Asynchronous callbacks (signal handlers, finalisers, memprof callbacks) can raise exceptions “out of thin air”; so-called interrupts (SML). This PR implements the masking of asynchronous callbacks which enforces critical sections in which asynchronous exception are guaranteed not to appear, as suggested in Dolan et al., Concurrent System Programming with Effect Handlers, §4.1.

The commits also have detailed logs; this PR is best read commit-by-commit.

1. Motivation

Currently, OCaml has two incompatible dialects: a dialect with asynchronous exceptions, very popular within a small community of developers and researchers of language tools (see #8873 (comment)), and a dialect with more-or-less reliable resource management (using unwind-protect-based abstractions when possible), popular within a small community of systems programmers. The dialects are incompatible because asynchronous exceptions make it currently impossible to reason about clean-up of resources, and both the requirements of asynchronous exceptions and resource management permeate whole programs. This work a pre-requisite to any attempt to reconcile the two dialects.

The problem is going to be worsened in OCaml multicore, which introduces a primitive notion of resource (fibers) that must be disposed-of explicitly and reliably: failing to do so (as of now) leaks the memory allocated for the fiber, and more importantly any resources "owned" by the fiber (i.e. freed via the exceptional termination of the thread). Asynchronous exceptions were correctly identified as a difficulty in Dolan et al.

The end goal (of which this is only a technical step) is to provide a resource-management and exception-safety model for OCaml which is robust in the face of asynchronous exception, at no added (developer, performance) cost compared to other kinds of unexpected exceptions and serious errors.

In this patch, we provide a way to switch for a short amount of time to a dialect that gives some C-style all-explicit control flow but where one accepts to be more careful about what one executes. The motivation for a dynamic switch is as follows: as a first approximation, one can imagine that this control could be provided by some static annotation on functions that instructs the compiler to remove opportunities for asynchronous callbacks to run. Allocations in particular would be replaced by new variants that do not run callbacks, similarly to what allocations in the C API currently do. In this first approximation, it would be possible to write safe resource-handling functions as desired, but at the cost of not being able to re-use existing code: one would have to write a non-polling variant of functions on lists, implement variants of runtime functions for arrays and system calls with a different polling behaviour, etc. It forces code duplication. To avoid this, we use a run-time switch understood by existing code, instead of static annotations. The role of a run-time switch is therefore to provide a polymorphism of polling behaviour for library code.

2. Overview

This PR includes a new per-domain state Caml_state->mask_async_callbacks, C functions to switch it (caml_mask, caml_unmask), a higher-order combinator (Sys.mask) in OCaml, and arranges the runtime so that it is easy to explore all polling locations for testing purposes. They are all meant for low-level use; the development of high-level abstractions (e.g. #8962) is outside of the scope. It introduces a distinction between possibly-raising and non-raising signal handlers, to exclude the Systhread tick from being delayed.

Optionally, it also includes a fix to a performance bug with C allocations when a signal is pending, and a combinator Sys.nonpreemptible_mask.

3. Design

A function Sys.mask is added, which guarantees that no asynchronous exceptions arise while it executes its arguments. It does so by delaying the execution of any possibly-raising asynchronous callbacks. Functions caml_{,un}mask are also provided for the FFI user. The tests demonstrates a toy implementation of an unwind-protect combinator safe of asynchronous exceptions (with_resource as proposed in #8962) inspired by GHC's bracket (it is essentially Fun.protect but adapted to protect against interrupts, for instance no exception should arise from resource release ever). Sys.mask can also be used on its own to solve issues such as presented in #8794.

The terminology mask comes from Reppy et al's work on interrupts in SML (https://people.cs.uchicago.edu/~jhr/papers/1990/cornell-tr-1144.pdf) and GHC (https://www.microsoft.com/en-us/research/publication/asynchronous-exceptions-haskell-3/). There is some delta between the paper and actual GHC. GHC has low-level scoped combinators mask (for interruptible mask) and uninterruptibleMask, that both provide an additional argument unmask, and which are then used to define higher-level abstractions such as bracket. We discuss the distinction interruptible/uninterruptible mask, and the unnecessity of unmask below.

Sys.mask needs sometimes to be used in combination with the polling behaviour of OCaml to provide useful critical sections (cf. the various comments (* BEGIN ATOMIC *) to see this combination in the testsuite). (The clarification of this polling behaviour is outside of the scope of this PR, see "Critical sections" at the end.)

We treat the systhread tick (a pseudo-signal which causes a context-switch every 50ms with system threads) as a special case. In principle, resource acquisition and release can call blocking system calls, in which case one wants to switch context (as performed inside caml_enter_blocking_section). So reacting to the systhread tick, which never interrupts the running thread with an exception, is always valid inside the mask. On the opposite side, I thought of the possible consequences of blocking the systhread tick inside the mask, while necessarily keeping context-switch inside caml_enter_blocking_section. Programmers might then be tempted to use Sys.mask to control the context-switching behaviour to ensure that no asynchronous or concurrent code runs at all (e.g. for races on mutable data). But this would be offering a dangerous interface for this job, given that it gives relevance to whether a function calls or not caml_enter_blocking_section (transitively), something that should remain an implementation detail permitted to change over time. The solution retained is to never block the systhread tick inside Sys.mask, and in addition to offer for discussion an optional Sys.nonpreemptible_mask better designed for that purpose (see below).

vs. unmask

By design, an unmask combinator as suggested in Dolan et al. is not proposed with this PR. Instead of unmask, the proof-of-concept with_resource is implemented by combining masking with OCaml's predictable polling behaviour. The latter ought to be strengthened and simplified with an explicit construct to control polling points, currently missing. But I have found that this technique makes reasoning about polling and resource safety easy, whereas there are things that I do not see how to do with an unmask combinator alone easily, like implementing the moving of resources in a way that is async-safe. Things like making a tail call after moving a resource are likely to be impossible with unmask on its own due to its scoped nature.

Non-raising handlers

As explained, we treat the systhread tick and blocking sections similarly since they are similar from a context-switching behaviour. Since the signal handler does not raise exceptions, it is allowed to run inside masks. This is implemented using a new notion of non-raising signal handler.

For backwards-compatibility reasons (avoiding to change the definition of Sys.signal_behavior), the interface to install a non-raising signal handler is not exposed via the Sys module, but remains available via a C primitive.

This could be left for explicitly undocumented low-level use, or one could decide on the contrary to have more kinds of non-raising callbacks that are allowed to run inside a mask in the future. (It could be desirable for instance to run finalisers that do simple release inside uninterruptible masks, since it would be realistic to write them carefully so that they do not raise, but an audit of finalisers in the wild has shown that changing this for all finalisers would be a largely-breaking change, so I did not explore this path.)

Non-preemptible mask

I propose for discussion the natural extension of Sys.mask to a variant Sys.nonpreemptible_mask, which disables all callbacks and context-switches for the duration of the scope. It offers the guarantee guarantees that no OCaml code runs asynchronously or concurrently. Accepting or not this part of the PR is matter of including or not the relevant commits.

One lesson of OCaml's cooperative concurrency libraries is that some people find it easier to control where context switches may happen, in the context of concurrency for IO. In addition, some OCaml code, for instance core's nano_mutex, take advantage of the absence of context switching and asynchronous code in allocation-free code to program efficient concurrency primitives. Sys.nonpreemptible_mask extends this capability to code that may allocate. On the one hand, avoiding allocations seems to have been sufficient for this purpose so far. On the other hand, OCaml programmers might have some experience of current limitations to report, or could find some other creative uses.

The programmer would rather avoid calling blocking calls inside Sys.nonpreemptible_mask, and the failure mode can be discussed. For simplicity, this PR unconditionally fails the call to caml_enter_blocking_section with an exception. While it is much better to unconditionally fail than to silently fail (by doing the context-switch), it is still desirable to keep an implementation detail the fact of whether a function transitively asks to release the runtime lock, and it would be possible to fail with an exception on some blocking calls whilst allowing calls to caml_enter_blocking_section in general without doing the context-switch. Depending on discussions, this would be done in a later PR.

vs. Interruptible mask

GHC also implements an "interruptible mask" in which asynchronous exceptions are masked but blocking operations may still be interrupted. The interruptible mask sounds like the appropriate choice for resource acquisition, during which, unlike resource release, exceptions are allowed if they can be expected (strong exception-safety), and one is likely to find blocking operations during acquisition that need to be interruptible.

However, this is more a question of library design for explicitly-interruptible blocking operations. During the work on EINTR-based signals #9722, we have seen that there are not many operations in the standard library that both explicitly poll on EINTR and then restart if no signal handler raises. For many blocking calls, running inside an uninterruptible mask will simply cause an Unix(EINTR,_,_) exception that the caller must catch, who can then decide to poll explicitly from OCaml. (This will require an explicit polling function that ignores the mask, not yet included.) For the remaining ones it might be appropriate to think of another interface to request explicit polling. This is less a question about the semantics of masking, and I prefer to postpone it to a later discussion (such as #8962) where it will still be possible to add an interruptible mask if it turns out to be needed.

(Thanks to @stedolan for an extensive discussion on this point.)

4. Implementation

When an asynchronous callback needs to run, action_pending in signals.c is set, and the runtime is notified via caml_notify_action. Notifying the program currently means setting young_limit to young_alloc_end so that the next allocation triggers the gc. To make masking efficient, one does not change how polling is done, but how the program is notified. Only if the new Caml_state->mask_asynchronous_callbacks is below the value of action_pending does the runtime notify the program. The program is also notified when unmasking if needed. Various levels of masking are provided, depending on whether they also delay non-raising callbacks.

More details are given in the commit logs.

Testing improvements

Testing the correct polling behaviour in OCaml is complicated. Tests based on sending signals from one thread to another are slow, brittle and nondeterministic. A few things are added to simplify testing, some primitives and notably some arrangements are made so that it is possible to re-record a signal from its own signal handler, allowing to explore all possible polling locations in a deterministic way.

Critical sections

The implementation of Sys.mask shows several critical sections ((* BEGIN/END ATOMIC *)) that cannot currently be enforced in bytecode and flambda. They are there to specify that the masking extends to the receiving and the returning of values, a fact described in the documentation. This is a lesson from implementing the with_resource combinator: Sys.mask works better if it combines well with non-polling sections on the caller side. But to observe this property, one needs to be able to enforce such critical sections in the caller's end in the first place. In backends where such critical sections cannot be enforced, there is nobody to observe the property. Therefore, the specification is vacuously true. Correct uses of Sys.mask in the meanwhile either carefully inline or solve simpler problems (e.g. #8994).

young_limit_c

An inefficiency has been noticed in the current polling mechanism in C, leading to the introduction of Caml_state->young_limit_c. This is mostly orthogonal to this PR. This is needed in this PR only due to the decision to not export action_pending, but the fix stands on its own to solve a problem that is already there. Godbolt has been used to check that the changes to the allocation macro do not affect generated code (more details in the commit log).