Would it still be a breaking change if, by default, dune used findlib only if $OCAMLFIND_CONF were set and kept using OCAMLPATH or whatever is available in PATH otherwise?

ability to understand toolchains defined in the same config file.

Dune used to fail unless variable(toolchain) = "value" was set in a separate file like findlib.conf.d/toolchain.conf. Was it ocamlfind's or dune's constraint? (UPD: Never mind, just read the commit comment.)

most would expect dune to use whatever configuration findlib is using.

Given the drop of direct dependency on ocamlfind in beta19, the whole behaviour is now actually undefined. 😁

Would it still be a breaking change if, by default, dune used findlib only if $OCAMLFIND_CONF were set and kept using OCAMLPATH or whatever is available in PATH otherwise?

Why would this be desirable? Dune should simply uses findlib whenever it is present. I assume that users who have findlib installed are assuming that its settings are being fully respected.

Given the drop of direct dependency on ocamlfind in beta19, the whole behaviour is now actually undefined. 😁

I think there's a bit of confusion here. Dune never used findlib via the ocamlfind binary. It has always used its own implementation of findlib to understand various findlib settings and interpret META files. The only reason we depended on findlib before was to avoid a race condition at installation time. Eventually, we found a better way to avoid the race condition, so we dropped the dependency on findlib. But we never used the ocamlfind binary even before beta19.

The issue in my opinion that dune is currently picking and choosing which findlib features it implements. This is why there are some limitations on things like defining toolchains in one file and allowing to set the libraries path via the ocamlfind config. In my opinion, dune should mimic findlib faithfully whenever it is available. Of course, there should be a way to turn this off.

Thanks for clarifying, it all makes sense now.

@rgrinberg can you think of a real scenario where this PR would change the behavior? The only I can think of is the following:

• you have a global installation of ocaml with a findlib.conf file that uses absolute paths for ocamlc, ocamlopt, ...
• then you do a manual build & install of ocaml and edit your PATH

with this PR, dune would still pick up the global ocaml rather than the manually installed one. However, absolute paths in findlib.conf don't seem to be very common. For instance, the findlib.conf file installed via opam install ocamlfind still causes the various tools to be looked up in the PATH:

$ cat `ocamlfind printconf conf`
[...]
ocamlc="ocamlc.opt"
ocamlopt="ocamlopt.opt"
ocamldep="ocamldep.opt"
ocamldoc="ocamldoc.opt"

Given that, I tend to think that we can reasonably expect that this PR won't be noticeable for users.

There's a couple of differences this PR brings (some are unrelated to each other):

• Cross compilation now works without findlib being installed. This seems useful for tools like esy that are planning to configure host/targets combinations using findlib.

• Toolchains are now defineable in multiple files without any requirement on file convention. I think defining multiple toolchains in one file is cleaner.

Additionally, if we were to go ahead with this PR, I would improve the emulation of findlib further. For example, path in findlib.conf can also be parameterized by the toolchain name. I think that's fairly useful as well.

I see. Personally I'm fine with these changes.

• you have a global installation of ocaml with a findlib.conf file that uses absolute paths for ocamlc, ocamlopt, ...

Does/will opam construct such a thing which people put in their global environment by way of sourcing the opam init?

If so, it would conflict with local esy builds when people also happen to have an opam environment initialized. If not, I think there's no worry (that I can think of).

If so, it would conflict with local esy builds when people also happen to have an opam environment initialized. If not, I think there's no worry (that I can think of).

Does esy set the findlib config explicitly anyway? If it does, then it has nothing to worry about.

The way to address the problem diml pointed out is by explicit configuration. The way I see it, dune respects the findlib configuration by default but should allow users to override it.

Using absolute paths in the findlib.conf will also cause problems if we build the compiler with dune and compose it in another build. I'm ok with having users manually override their findlib installs if those are not properly configured.

@rgrinberg Esy will set findlib config explicitly as soon as we merge it (we will temporarily set OCAMLPATH as well for older dune to keep working).

Looking at the current code, whenever we have opam installed, then we use it to discover where the libraries should be sourced from rather than $ ocamlfind printconf path. That seems a bit off to me and we should change it to respect the findlib configuration. Should we add a Build_environment_kind that reflects a findlib based setup. Opam2/Opam1 can be used whenever no findlib.conf is given.

It does seem like we have too many entry points for the user to configure things, and it is starting to become hard to understand how all these interact between each other.

Rather than fiddle around, I suggest to start a discussion with the authors of the various tools to see if we can reach an agreement. If we can agree on a blessed way to setup the environment for OCaml compilation, then we don't need to support every possible combination.

Sure, I agree with that. I just thought that findlib was basically that standard already. Let me cc @andreypopp here as well. If there's a consensus from the esy team on what exactly dune should support to configure compilation environments, that would be an important step forward already.

Well, that's almost true. Over the years, findlib has mostly become the standard. There is however one catch: ocamlfind is not mandatory and is not always installed. This is a big issue when you have several installations of OCaml, such as a system one, an opam one and/or an esy one on top.

If we agree that variables such as OCAMLPATH have authority and specify the ocaml installation to use, then that's fine. However, all the various tools and in particular opam must set this variable consistently.

There is still an issue regarding findlib.conf: currently the only way to locate it is to call ocamlfind printconf conf. However, the ocamlfind binary that is in the path might not be the one from the right installation. For instance, if you have an ocamlfind installed via your distribution, and you are using opam but didn't install ocamlfind, then ocamlfind printconf conf won't report the right configuration file :/

There is still an issue regarding findlib.conf: currently the only way to locate it is to call ocamlfind printconf conf. However, the ocamlfind binary that is in the path might not be the one from the right installation. For instance, if you have an ocamlfind installed via your distribution, and you are using opam but didn't install ocamlfind, then ocamlfind printconf conf won't report the right configuration file :/

Okay, I'm convinced that we shouldn't use findlib as the default for this reason. On the other hand, I think we should go for full findlib support whenever OCAMLFIND_CONF or OCAMLFIND_TOOLCHAIN are set. I think the possibility for these variables to be set accidentally is quite low.

I think we should go for full findlib support whenever OCAMLFIND_CONF or OCAMLFIND_TOOLCHAIN are set.

This would be most preferable.

@jeremiedimino I'm wondering if we should revive this PR but modify a couple of things. We'll just respect OCAMLFIND_TOOLCHAIN and OCAMLFIND_CONF and completely ignore the ocamlfind binary otherwise. To me this seems like the most consistent way of dune working. I don't quite like the current behavior that we got rid of ocamlfind in the regular path, but still keep it around for cross compilation.

Agreed that the current state is a bit wonky. Why do we need to support OCAMLFIND_TOOLCHAIN and OCAMLFIND_CONF at all?

For cross-compilation, it feels like we could just look for $OPAM_SWITCH_PREFIX/lib/findlib.conf.d/<target>.conf and be done with it.

I wonder if it is not used for cross-compilation in distribution. For example ocaml-mingw-w64-x86-64 defines an etc/x86_64-w64-mingw32-ocamlfind.conf. However I haven't found how it is used. Moreover we could find a simpler way to support this use case with dune.

Ah, I wasn't aware distributions also installed such a configuration file. One idea would be to allow to configure a pattern for such configuration at compile time. That would be in-line with the optional compile-time configuration of the library path.

Agreed that the current state is a bit wonky. Why do we need to support OCAMLFIND_TOOLCHAIN and OCAMLFIND_CONF at all?

To make it easy for opam-cross maintainers to adapt. IIRC, they still have non dune packages.

For cross-compilation, it feels like we could just look for $OPAM_SWITCH_PREFIX/lib/findlib.conf.d/.conf and be done with it.

Let's see if @toots is fine with that.

Ah, I wasn't aware distributions also installed such a configuration file. One idea would be to allow to configure a pattern for such configuration at compile time. That would be in-line with the optional compile-time configuration of the library path.

That seems good.

Cross compilation is pretty tightly tied to opam so I believe, as long as we honor OPAM_SWITCH_PREFIX we should be fine since it looks like opam is currently setting it.

As a side note, we're pretty much stuck with updating dune for cross-compilation until #3751 is merged/fixed.