@rgrinberg I am trying to add this new benchmark with a synthetic project, but dune seems to choke when building it, see sample build (already running for 2+ hours): https://github.com/jchavarri/dune/actions/runs/4281888888/jobs/7455464113

One can replicate locally following these steps, assuming you have the branch checked out:

cd bench
dune exec ./gen_synthetic.exe -- -n 6 test
dune exec ./test/main.exe

It remains stuck on Saving digest db... and never gets over that step. Do you have any hints what could be going on?

These are the last syscalls that i can see before killing the process, produced with strace -rT -o strace.txt _build/default/bin/main.exe exec ./bench/test/main.exe:

     0.000103 brk(NULL)                 = 0x55fe7d9e6000 <0.000010>
     0.000030 brk(0x55fe7da07000)       = 0x55fe7da07000 <0.000010>
     0.000088 mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9bf3f23000 <0.000010>
     0.000180 mmap(NULL, 2105344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9bf3d21000 <0.000010>
     0.000036 mmap(NULL, 1024000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9bf3c27000 <0.000010>
     0.000048 sigaltstack({ss_sp=0x55fe7d9fd480, ss_flags=0, ss_size=8192}, NULL) = 0 <0.000011>
     0.000037 rt_sigaction(SIGSEGV, {sa_handler=0x55fe7bcf5b30, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_NODEFER|SA_SIGINFO, sa_restorer=0x7f9bf414e140}, NULL, 8) = 0 <0.000012>
     0.000041 readlink("/proc/self/exe", "/home/me/code/dune/_build/defaul"..., 256) = 53 <0.000031>
     0.000084 stat("/home/me/code/dune/_build/default/bench/test/main.exe", {st_mode=S_IFREG|0555, st_size=4372552, ...}) = 0 <0.000021>
     0.000059 brk(0x55fe7da30000)       = 0x55fe7da30000 <0.000011>
     0.000036 lseek(0, 0, SEEK_CUR)     = -1 ESPIPE (Illegal seek) <0.000010>
     0.000033 mmap(NULL, 794624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9bf3b65000 <0.000010>
     0.000032 lseek(1, 0, SEEK_CUR)     = -1 ESPIPE (Illegal seek) <0.000009>
     0.000030 lseek(2, 0, SEEK_CUR)     = -1 ESPIPE (Illegal seek) <0.000011>
    10.531237 --- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
     0.000497 +++ killed by SIGINT +++

it seems an issue with too long args for linker. Edit: this was not the case.

This is ready to review now.

I did not manage to solve the issue with dune hanging when trying to execute the binary, but I ended up using dune build which works and is actually better because we want to track build performance, not execution. I will open a separate ticket for that mysterious problem.

The new benchmark results can be seen in https://jchavarri.github.io/dune/dev/bench/. Note the first commit in the chart had too short build time due to dune clean not being called properly.

The only remaining questions would be about:

• is the current amount of modules in the synthetic benchmark (1296) enough? or having a project with more modules would be desirable for some reason?
• it is possible to measure incremental build times in a separate benchmark, would this be interesting to measure somehow?

Do you think we can simplify the script a little?

1. I don't think we need multiple libraries. You should be able to show the regression with 1 lib
2. A single directory should be enough as well

In summary, the only parameter we should need is the number of modules to generate

Simplified gen_synthetic in 7de1cbd.

I only removed the subfolders, because the generated code was only using one library already.