@diml have a look at how I've implemented versioning. I'm not entirely confident by how things are done right now on the encoding side as there's quite a bit of boilerplate. I suppose it's because we don't have any combinators to output a list of sexp, rather than just 1 big record.

@diml Hilariously, not having #966 is blocking our plans for the META -> Dune_package.t conversion that we planned. Our Dune_package.Lib.dir function assumes that every sub library has its own sub directory, but this isn't true for the stdlib (unfortunately). We can either:

• bring back the old legacy Findlib.Package.t -> Lib_info.t which already takes care of this special case.

• Add a dir field to Dune_pacakge.Lib.t and then special case the stdlib conversion.

Let's do the second option, it seems more flexible.

@diml I chose the 2nd option. As a consequence, the dir field must now be Path.t as opposed to Path.Local.t. The paths that we get from reading the META files aren't necessarily local, so we must treat them more generally.

Ok, makes sense

Hm, this path is a bit more complicated than I thought. All of the various artifacts fields:

    ; archives         : Path.t list Mode.Dict.t
    ; plugins          : Path.t list Mode.Dict.t
    ; foreign_objects  : Path.t list
    ; foreign_archives : Path.t list Mode.Dict.t
    ; jsoo_runtime     : Path.t list

Seems like they should be Path.Local.t rather than Path.t when encoding the paths. As we only want to encode the bit that is relative to the dir field. But it's quite a bit more inconvenient to deal with Path.Local.t for artifacts, plugins, etc. I'm wondering if we should one of the following:

• Change the Path.t for all of these library relative paths and make this record abstract. We'll add accessors functions such var archives : t -> Path.t list Mode.Dict.t that will concatenate the paths appropriately.

• Add a path parameter for this Path.t type. When encoding, this should be Path.Local.t. While when decoding, it should be Path.t

@diml when printing the (lang dune x.y) line for the dune-package file, which version do we use? Do we use the latest version? Or the version in the dune-project of the package. The latter is harder, as we'd need to support serialization to different versions. But at least this way, users would have a controlled way of upgrading the package format.

Let's use the latest version. Using the one specified in the dune-project is much more work, and the only advantage I can think of doing so would be that the user could downgrade dune without reinstalling existing packages. This is not likely to happen very often so it doesn't seem worth the effort. Can you think of any other advantage?

I don't see any advantages, but I still think that it will cause some surprises to users. Let's prepare the team for this decision as well btw. I'll mention this to everyone on slack.

That seems OK to me, but I'm fine with waiting for #966 as well. And in the meantime, we could use some heuristics to detect ppx used as libraries.

Unfortunately, it's not clear whether the OCaml PR #966 is waiting on will go in or not. So for now we should assume it won't go in. Additionally, the new layout would only apply to the latest version of the compiler. We still need to support existing versions of compiler, as well as OCaml packages that don't use dune.

Let's use the latest version. Using the one specified in the dune-project is much more work, and the only advantage I can think of doing so would be that the user could downgrade dune without reinstalling existing packages. This is not likely to happen very often so it doesn't seem worth the effort. Can you think of any other advantage?

Can't think of anything. Let's just use the latest version. The other devs don't seem to mind either.

I've rebased this PR on top of master. There's still some hiccups with the paths unfortunately. A naive conversion like:

val to_dune_lib : lib -> Dune_package.Lib.t

Will not work. This is because we'd like the paths in the Dune_package.Lib.t to reflect the path structure in _build/install. The solution I'm working on is to change the signature to:

val to_dune_lib : lib -> map_paths:(Path.t -> Path.t) -> Dune_package.Lib.t

map_paths can be easily generated after running the copy rules from the object dirs to the install dirs. Another approach would be to have an explicit function that would simply change the root dir for a Dune_pacakge.Lib.t. But this doesn't work as well because some paths have a completely different structure when stored as installed paths. For example, the cma's live in the obj sub dir of a library in _build/default, but there's no obj sub dir in the corresponding _build/install.

What about using the following for a library with public name "a.b.c":

Path.relative (Config.local_install_lib_dir ~context ~package:"a") "b/c"

?

So take all the paths in Dune_package.Lib.t, take their basenames and prepend them with the prefix you've proposed?

I think that will work currently but unfortunately, it will flatten all the objects into 1 dir for installation. That should work well now, but it might be break if we'll need a subdirectory for some files in an installed library (This could happen if we'll have some cmi's that we only share between libraries for example).

I'll give that a try, but it might still be used as the argument passed to map_paths:(fun p -> Path.relative lib_install_root (Path.basename p)).

Note that this corresponds exactly to the directory structure we use in install/..., so it should be enough.

In the last commit things are starting to work. I think that we'll need to update the subsystems API if we'll want subsystems to install their own artifacts.

Okay, this PR is now ready for review. There's still a little more work to do to make these files work for virtual libraries but that can be done in subsequent PR's.

Sorry for the delay, I feel like this PR needs careful review but I'm a bit busy at the moment. If I don't get a chance to review it by Thursday, we can look at it together then.

Agreed. This is why I didn't want to push this into 1.6 either. It would suck to have to many versions of this format.

Here's a summary of the discussion with diml about this feature:

In anticipation of dune 2.0, we will eventually want to have a form of forward compatibility for dune-package files. We'd still expect dune 2.0 to be able to generate valid dune-package 1.x files. @diml can explain the reasoning for this, as I've forgotten why can't we have dune 2.0 just generate dune-package files valid for 2.0.

We'd like the file paths in the dune-package file to be relative to the dune-package itself. The alternative was to infer the full file paths from the library names.

Subsystem (plugin) handling requires that dune knows which fields and stanzas to ignore when it doesn't recognize a plugin/subsystem. We've decided handle this as follows:

Every declared plugin will have an appropriate (using foo x.x) in the dune-package file. Such a plugin will only be able to define foo or foo.* stanzas and fields. If dune doesn't recognize a declared plugin, it will ignore the appropriate stanzas/fields. cc @ejgallego regarding coq.

For variants support, we need to add some extra module information. There's a couple of possibilities on how to handle this:

• Just include the module information in the dune-package file (might as well do it for every library in this case). However, including information in the dune-package file should be done with care. As it means that we'll have to support this format forever.

• Try to infer it by scanning the file system. While we won't have to change the dune-package file with this approach, we'll have to add rules based on the results of evaluating some globs (to infer modules). This isn't very desirable either.

Personally, I'm leaning towards the former.

The last commit makes subsystems in the dune-package file only generate using the dune syntax.