use DUNE_CACHE_HOME instead of XDG_CACHE_HOME#11612
Conversation
b469794 to
8894b32
Compare
|
I just rebased on main, fixed the conflicts, and addressed your comment @Leonidas-from-XIV. |
|
I am generally not sure whether this is the right thing to do. This is because what we call "the dune cache" is a build artifact cache (and its location is controlled by Thus for consistency to me it would make more sense to make |
1d90d4b to
7efb83d
Compare
I think I understand where you're coming from, however wouldn't that be breaking? If someone has set To recap:
I hope I got that right. I was of the opinion that the breaking option would be better before making that table, now I'm in the 'compromise' camp. However I think having more env vars can be detrimental in the long run, especially if they're vaguely named (for example, the comment above wrote Thoughts? |
|
I think the idea of introducing a Removing If Finer control for the other kinds of caches dune creates i.e. toolchains and the lmdb cache can be added if the need ever arises, but I am against adding that now. Those should just live in their subdirectories in |
|
Thanks for making the table @ElectreAAS, it is a great overview! Once implemented that explanation could even go into the documentation because it explains which locations and environment variables are used when.
Looking through the table, I also find myself nodding at the things proposed in the "compromise" column. It seems fairly reasonable and allows to set all caches to a specific location at once (and maybe In particular I believe there are two no-go's for me:
The "compromise" solution seems to address all my concerns. I also agree with @Alizter that keeping the build cache configurable needs to stay but the other caches can for now just be controlled via |
9a17010 to
9243be2
Compare
|
I've went ahead and implemented the 'compromise' option. We now have DUNE_CACHE_HOME & DUNE_CACHE_ROOT, and tests should be correct by now. |
There was a problem hiding this comment.
Note: I haven't changed the semantics of dune cache clear so it would also delete the revstore & toolchains cache, but that's debatable
There was a problem hiding this comment.
That's a good question. In theory it changes the behavior but I guess it does make sense that dune cache clear removes all the cached data. Before we didn't have a way to clean out the rev-store nor the toolchains so I'd argue that this is an improvement.
One could extend the command to allow clearing only selected caches, but I don't see an urgent need for that, so I'd be willing to wait for users actually requesting this.
There was a problem hiding this comment.
Any kind of change to the caching command seems like it should land separately in any case.
Leonidas-from-XIV
left a comment
There was a problem hiding this comment.
It's just a preference but I think it would make sense to keep the different types of caches in their respective places and not introduce dependencies on each other (especially as they aren't functional dependencies, just file paths).
I'd rather suggest creating dune_cache_dir or so to contain the logic and then Dune_cache_storage.Layout, Dune_pkg.Rev_store and Dune_rules.Pkg_toolchain can use this.
|
This PR is still marked as fixing #11585. Is that still the case? |
Ah this was an edit by someone else that I didn't catch. No I don't fix that one. Adding |
|
If you update the PR description and importantly remove the |
|
Oh, my apologies. It was me who added it, I thought that |
|
In the meeting we discussed perhaps biting the bullet and changing the definition of Currently we are not aware of any actual breakage, but it will mean that users existing caches will be invalidated. This is already the case when we change action digests so perhaps isn't a big deal. If we change the meaning, then a user may have This has the advantage that setup-ocaml will not need to change the variable, we don't introduce cruft and nothing is broken. |
3333d18 to
cf3d159
Compare
That would mean implementing the 'Breaking' option instead of the compromise one, right? |
Yes, it would be that proposal. However we should be clear that we don't actually know if it causes any breakages other than missing old caches. I think a separate PR would be good and we can weigh the merits of both. |
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
…dune_util` Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
cf3d159 to
d794acd
Compare
|
#12858 has been opened with the 'breaking' option |
…caches (#12858) This PR is an alternate version of #11612, implementing the 'breaking' option discussed [here](#11612 (comment)) It fixes #11584, and helps (but does not fix) #11585. --------- Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
|
Closed in favor of #12858, which implements the 'breaking' option discussed above |
…caches (ocaml#12858) This PR is an alternate version of ocaml#11612, implementing the 'breaking' option discussed [here](ocaml#11612 (comment)) It fixes ocaml#11584, and helps (but does not fix) ocaml#11585. --------- Signed-off-by: Ambre Austen Suhamy <ambre@tarides.com>
CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the sandbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)
CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Fix package extraction on systems with tar implementations that don't auto-detect compression (e.g., OpenBSD). Dune now passes explicit decompression flags (-z for gzip, -j for bzip2) when needed, and provides clear error messages for unsupported formats like XZ and LZMA. (ocaml/dune#13688, fixes ocaml/dune#10123, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)
CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Fix package extraction on systems with tar implementations that don't auto-detect compression (e.g., OpenBSD). Dune now passes explicit decompression flags (-z for gzip, -j for bzip2) when needed, and provides clear error messages for unsupported formats like XZ and LZMA. (ocaml/dune#13688, fixes ocaml/dune#10123, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)
CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)
Fixes #11584 and should ease the pain of (but does NOT fix) #11585.
The build cache is currently set up to be at
$DUNE_CACHE_ROOT, which itself will default on$XDG_CACHE_HOMEif not present.Three different caches skipped
$DUNE_CACHE_ROOTand used the XDG directly: the toolchain cache, the git-repo cache, and the lmdb (rev store) cache.This PR introduces a new variable:
$DUNE_CACHE_HOMEto allow customization of cache locations without breaking existing behaviour. To recap:DUNE_CACHE_HOMEdefaults toXDG_CACHE_HOME/duneDUNE_CACHE_ROOTdefaults toXDG/dune/dbDUNE_CACHE_HOME/dbDUNE_CACHE_ROOTorXDG/dune/dbDUNE_CACHE_ROOTorDUNE_CACHE_HOME/dborXDG/dune/dbXDG/dune/git-repoonlyDUNE_CACHE_HOME/git-repoorXDG/dune/git-repoXDG/dune/rev_storeonlyDUNE_CACHE_HOME/rev_storeorXDG/dune/rev_storeXDG/dune/toolchainsonlyDUNE_CACHE_HOME/toolchainsorXDG/dune/toolchains