Hi,

I'd like to have your opinion on how to configure dune projects.

The problem

Some pieces of software require a build-time configuration. We already have configurator to query for some compile flags, or whether the host system supports certain features.

This is a useful, but not enough in certain cases. Let us see two examples:

nocrypto configuration

Nocrypto is a library of cryptographic primitives. Some of them are implemented in C, and have two versions: a generic, portable one, and an optimized one. The optimized one uses compiler intrisics that require a compatible compiler and CPU.

To select the implementation, it uses the cpuid library whether the host system supports it, and generates corresponding compilation flags.

That corresponds to the following code (N.B.: this is unreleased code):

(library
 (public_name nocrypto)
 (libraries ...)
 (c_names ...)
 (c_flags (:include cflags.sexp))
)

(rule
 (targets cflags.sexp)
 (action (run ./discover/discover.exe --output %{targets})))

; in discover/dune
(executable
 (name discover)
 (libraries dune.configurator cpuid ...)
)

Where discover.ml outputs the cflags depending on what cpuid returns.

This works, but it is missing an important feature: how to override the detected value? That can be useful, either to build a portable binary on a machine that supports acceleration, or to force an optimized binary and fail hard if this is not available.

Frama-C's share directory

Some programs make use (at runtime) of data files in a special directory; that can be configuration files, binary assets, scripts, etc.

This is in particular the case of Frama-C. Before its compilation, a configuration step defines a set of paths that are expanded in the OCaml source code. Data files are resolved using this macro expansion.

The ideal case is to make the binary relocatable and not depend on hardcoded paths, as described in #1185 (this can be implemented by using relative paths between the binary and the data directory). But that is not always easy. In particular, existing projects that use an explicit configuration step for this cannot easily be ported to dune.

How to fix this?

I propose to add a new kind of variable: project variables. (N.B. syntax is to be discussed separately).

Project variables are expanded in rules using %{project:name}. For example, in the above nocrypto rule:

(action
 (run ./discover/discover.exe
  --accelerate %{project:accelerate}
  --output %{targets})))

Project variables need to be set in dune-project:

(using project-variables 1.0
 (variables
  (accelerate detect)))

Each variable has a default value; the string "detect" in this case. If nothing in particular is done (such as when installing from opam), project variables are expanded to their default value.

But it is also possible to call dune configure set accelerate true: after doing so, the variable would expand to the string "true". This state is persisted through a file under _build. This is tracked as a dependency, like environment variables are. In particular, configuration changes will trigger a partial recompilation (conservative but hopefully precise).

Revisting the nocrypto example, this would mean adding a --accelerate argument to the configurator configuration, that would parse "false", "true", or "detect". If the latter is passed, it would call the detection function. Some support for that could be added to configurator itself. A similar exists in topkg as discovered_key.

As for Frama-C, this could enable a data-dir project variable. It is a bit more tricky to implement, because its default value would depend on the context's opam configuration. So, the exact details are more complicated, but reasonable semantics can probably be defined. An alternative is to have predefined project variables like prefix that can be interpreted by dune install.

It seems likely that this feature would work well with vendoring tools like duniverse: if we allow overriding variables from all projects in dune-workspace, we could reconfigure dependencies without having to patch their sources.

cc @bobot @avsm @samoht

Thanks!