I just wrote my first cram test and encountered some minor usability issues, reported below. (Let me know if I should split them in three separate issues.)

Git error

The first is very minor: running the test (before it has been promoted to include proper output) shows a rather scary

         git (internal) (exit 1)

near the top of the output. (git and (exit 1) are in red.) This is confusing as it looks like cram tests interact with my git workflow somehow, when I don't expect them to. There is a grayed-out line right below

(cd _build/.sandbox/91100af1cb755fb629459b1f7df1f2d7/default && /usr/bin/git diff --no-index --color=always -u ../../../default/bin/test_errors/json.t/run.t bin/test_errors/json.t/run.t.corrected)

from which I can actually understand that you are using git diff as your diff utility, so the error is not related to an interaction with my git repository. Still, the first message, emphasized as an error, is unnecessarily confusing, and it would be better to not show it. Maybe a proper error message like: "Test output mismatch, see foo.corrected" could be used instead.

Hard-to-enable cram tests

To enable cram tests (to check error display in a binary), I created a new test_errors directory with the following stanza:

(cram
 (deps %{bin:mustache}))

Then I wrote my first cram test (without putting the correct output, so I know it would fail), and I ran dune runtest, and then nothing happened:

Done: 0/0 (jobs: 0)

I assumed that there was something wrong with my cram syntax, so I tried various variations (single-file test instead of directory test, two-spacing of commands or not, etc.); each time Dune would do absolutely nothing.

Then I realized that I had forgotten to include (cram enable) in my dune-project file.

This is a usability bug. I'm not sure if Dune should show a warning if .t files occur without cram tests enabled (this sounds like a usability disaster for a project using .t extensions for something else), but it should definitely say something if I use a (cram ..) stanza in a dune file, without cram tests enabled.

Not-so-clear documentation for cram directory tests

When you create a directory-style cram test, should run.t commands be 2-indented like in normal cram tests, or should they start at the beginning of the line? (Sounds like a silly question, but it's the sort of question you ask yourself when your test is not doing anything and you don't know why.)

According to the current documentation, commands in run.t in directory tests should not be 2-indented:

We may then
access their contents in the test script ``run.t``:

.. code:: bash

   $ wc -l foo | awk '{ print $1 }'
   4
   $ wc -l $(ls bar) | awk '{ print $1 }'
   1231

We may then access their content in the test script run.t

$ wc -l foo | awk '{ print $1 }'
4
$ wc -l $(ls bar) | awk '{ print $1 }'
1231

According to my testing, this is incorrect and 2-indentation should be used. (Which is more consistent with single-file cram tests, and more convenient to write comments.)

Specifications

• Version of dune: 2.7.1